AppEngine Dataflow SpringBoot In a past blog I wrote about Scheduling Dataflow pipeline. There I described how to leverage google's app engine and cron server to schedule dataflow pipelines. For simplicity I had used java spark as the web server. Thought now I have decide that spring boot is more...
Design and implement an ETL pipeline to stream data from salesforce to Bigquery using apache beam on top of google dataflow. The task included understanding the amounts of data, and the api’s that salesforce supports. The delta changes needs to be transferred every 15-30 minutes across all tables, with configuration options to add tables and columns in the future without code changes.. Apache Beam on top of dataflow was chosen as the data pipeline for it’s abstract sdk for processing data. Google App-Engine was used as the scheduling mechanism, and google app script as the monitoring dashboard.
Help a startup to re architecture a monolithic application to a micro-service platform. This included breaking up application to multiple services with kafka as the communication bus. Help redesign application for scale by introducing rx-java in the software tier along with clustering of servers, and cassandra as a backend database for scaling data. Help introduce more functional style programing using javaslang framework On the devops side, I helped create the Cloud-formation structure on top of AWS for clustering of servers
Architect for application that downloads and parses documents from multiple sources. /n Support for big load using Spring Boot architecture in the Scala language on top of Akka. /n Application was then packaged in Docker. Tech leader on legacy product written in scala for text NLP analysis. /n Add framework for streaming data to application lucene indexes from server databases.
Tech leader on monitoring team of Amdocs Portfolio. /n My job was to bring the product to next level including enhancements in cluster definitions using zookeeper. /n Introduce and create framework for sending dropwizard metrics to graphite using pickel protocol for optimizations. /n Introduce elasticsearch to product for event analysis. Enhance metrics library embedded in portfolio by creating dynamic proxies for metrics.
Joined a team that just started to build a Spring/Hibernate product on a legacy DB./n Helped as tech lead to solve multiple hibernate issues including nested filters and dynamic cache of metadata not supported by the hibernate model. /n Introduced Spring Integration to team, with full flow using JMS, xml transformations, and web service calls (based on classic integration patterns).