Yitzchak Lieberman

Java Big Data  @ Nurego

Nurego is a startup on the area of IOT biz-ops which was acquired by their largest customer - GE My role was to help them build from scratch BI infrastructure for querying and analysis of their usage data by their customers. I was participating in the POC of deciding which technology to use (go on open source Hadoop with Hive/HDFS or purchase on the shelve product HPE Vertica), then building mechanism for synchronizing metadata tables from DBMS (PostgreSQL) to Vertica (include tests), Build a docker container for Vertica with third party packages for connecting to PostgreSQL and prepare API service for use of UI dashboard (and other purposes - customers requirements and public api of Nurego metering service) The technology stack consists of: Vertica - persistence layer for usage data and metadata. PostgreSQL - persistence for metadata of Nurego main service. Java Spring boot with hibernate - for api service Docker and docker-compose - containers for services and DBs. Git, Maven and Jenkins - builds, CI and deploy tools. Kibana - monitoring and log aggregation.

Java Big Data  @ Amdocs

Amdocs: srefactoring legacy product in order to enhance its runtime performance analyzing, planning and developing the solution by defined and modular mile stones. Technology stack : java 8, jdbc, oracle DB, maven, jenkins, jprofiler, amdocs proprietaries - perforce VCS, java libraries

Java Big Data  @ BigID

BigID is a startup that helps organizations to map their data sources and prevent from sensitive data (PII data - ssn, id numbers, etc.) to be used by unauthorized systems (client product is in demo and POC phase).

• My role was to help them with integrating and implementing connectors to Hadoop services (HDFS, Hive), Amazon services (EMR, S3, EC2) and scan them in efficient way, in addition to help improving development workflows, adding integration tests to cover common scenarios and giving my opinion on architectural issues regarding product framework and development life cycle. Tech stack
• Hadoop HortonWorks vendor - HDFS, Yarn, MapReduce 2 .
• Amazon services - EMR, S3, java sdk.
• Node.JS - some of BigID services are written in JS and running on Node.
• MongoDB - for identity and scanning metadata persist.
• Docker - all BigID services running in container of Docker.
• Apache Tomcat - Scanner service container.
• MySql and Postgres - identity source of PII records.
• AngularJS - UI main application and screens.

Java Big Data  @ Vox populi

The project was to develop an app with back end platform for second screen world. back end is micro services based with persistent layer of Redis and Couchbase and pub/sub messages of Kafka, and also storm for complex Key Accomplishments:

• application would serve registration, joining show episodes and voting and back end services for registration, calculating votes counters and ranking and api’s for the television show control room.
• The tasks were on all back end micro services - fixing bugs, design and implement social login flow process, evaluate couchbase performance on indexing, redesign and implement of storm topology, etc.
• My main part on the organization was to design and develop the social login flow that will allow users to sign in on our app with their facebook, twitter or google accounts. also i’ve touched the storm topology for adding new stream of data for votes with wrong answers and helping on design how to persist all votes in order to process 100% of them.
• On the client side it is applications for Android and iOS, server side it’s combined from kafka, storm, redis, couchbase, spring boot micro services architecture,Vert.x, hibernate. my part was touching all server side stack, only on Vert.x i touched the least. also we have started integrating elastic search for querying over couchbase data.
• Storm for processing votes counters and ranking, kafka for sending commands and votes, couchbase for persisting users data and votes, redis for persisting temporary votes counters and users state events, elastic search for querying data.
• A bit on storm consistency and data loss, end to end system durability and on social login flow and what way to detect users that were logged in in the past or from other devices.