Apache Beam Workshop

How to write TrafficMaxLaneFlow pipeline and deploy it on the google dataflow platform.

In this workshop we will write together a pipeline that will reads and analyses data from traffic sensors.

The TrafficMaxLaneFlow pipeline analyses data from traffic sensors. This pipeline can run in both batch and streaming modes. In batch mode, the pipeline reads traffic sensor data from an input file. In streaming mode, the data constantly flows in from a Cloud Pub/Sub topic.

TrafficMaxLaneFlow analyzes the incoming data stream using Windowing, specifically Sliding Time Windows. Sliding time Windows use time intervals in the data stream to define bundles of data with windows that overlap.

TrafficMaxLaneFlow uses a custom Combine transform to extract lane information and calculate the max lane flow found for a given station for each Window. A custom combine transform is necessary because the combination is not a simple Max combination it needs to retain some additional information along with the flow value.

The pipeline formats and writes the max values along with the auxiliary information to a BigQuery table.

by Chaim Turkel, Java Architect

Call us    +972 (3) 6488618
Thank you for your interest!

We will contact you as soon as possible.

FOR MORE DETAILS CONTACT US

Please leave your details and we’ll get back to you as soon as possible

Oops, something went wrong
Please try again or contact us by email at info@tikalk.com
Thank you for your interest!

We will contact you as soon as possible.

Send us a message

Oops, something went wrong
Please try again or contact us by email at info@tikalk.com