Introduction I would like to post a short description about a simple design change, I just did for one of Tikal’s customer, which greatly improved the throughput for their processing on their BigData lake with Spark. Background In the last few months I had to build a BigData infrastructure for...
My main goal here was to architect and build Big-Data Platform for “Worldmate”. The platform was based on Lambda-Architecture, using various Big-Data technologies like Spark, Kafka, Logstash, Hive and more, as well as AWS services (Kinesis, S3, EMR, RDS), since the cluster was running on AWS EMR. The platform target was to consolidate Worldmate various end-users data : Business Traveler Data. This data includes flights-booking, hotel-booking and travelers-trips, located in various data sources (i.e. flat-files, RDBMS, MongoDB etc), and provide a platform for Machine-Learning models and BI reports tools (i.e. Tableau –Server). I wrote Spark applications (in Scala), tried machine-learning models in Spark (Collaborative-Filtering, Frequent-Item-Set), and also built travelers graph, using Spark’s Graph-Frames and Spark’s GraphX, to create social associations between travelers, to predict traveler next flights and apply “Team-Travel”.
In this task I created architecture and implementation “from scratch” for a new fintech start-up company named “Sunbit”. Sunbit vision was to enable splitting a purchase into multiple payments - providing customers the benefits cost efficient payments. My role was to create an architecture for this idea, and make it happen. I used mircoservices technologies (i.e Spring-Cloud, Spring-Boot), developed the business logic, including various perspectives like security and resilience, and brought it into a live AWS services. As part of the process, I created an “Architecture-Description” document for the product, and then quickly started to implement its skeletons. I also used Spark and machine-learning to build a machine-learning “model”, which provided a prediction engine service to the other services in the product.
Architect and consultant to scale out an existing application by changing its architecture. The application is “GPS analytic application”, and in order to support the increasing end-users capacity, I introduced reactive technologies (Vert.X), as well as messaging system (Apache Kafka) and caching system (Redis). I wrote the new service from scratch using Java8 with Vert.X (running as a separate micro service), refactored the exiting SpringBoot anaytic application to support the change, by applying all phases for both - Going from design and development, then to performance testing, and last into production on AWS.
Initiating a new product for a new startup, which aims to anaylze SaaS cloud providers regarding security and file content management. I was building the architecture followed with a POC that scans organizations information on SaaS providers (i.e. GoogleDrive and Box.com). This scan includes users , files and events for these organizations, in order to provide content base knwledge for them, using the SaaS providers platforms. The POC is deployed at AWS and uses Spring platform (including Spring-Social, Spring Security, Spring-Data and more).
Architecting a new service product (running on Amazon), which servers as a portal of electronic-components for Mentor’s users. The product is a Spring/Hibernate web application running on Tomcat, which includes text-search engine that enable users to search and store electronic components from multiple suppliers, and use the stored parts in Mentor tools. In addition, I implemented the model, persistence (Hibernate, Solr) and also various services needed for the service (i.e. security, transaction configuration using Spring).