Streaming Data Pipeline
Doing It The Right Way

Creating near real-time data pipeline for billions of events and terabytes of data can be a challenging task. How can we serve the requests, with low latency and persist them all in less than a second latency end-to-end? What are the do’s and don’ts when rolling such a system to production and how to monitor the hundreds of components involved? Come to hear a real-life case of such a system.
We will discuss the architecture, technologies, code and DevOps concern of the streaming data pipeline we created for one of the top ten mobile games in the US.

by Haim Cohen, Big Data Architect & Machine Learning Engineer