Dataflow for stream/batch processing in GCP - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is Google Cloud Dataflow?

Google Cloud Dataflow is a fully managed service for processing and analyzing large data sets in real time or batch mode. It helps you build data pipelines that can handle streaming or batch data easily.

beginner

What is the difference between stream processing and batch processing in Dataflow?

Stream processing handles data continuously as it arrives, like watching a live video. Batch processing handles data in chunks or groups, like watching a recorded video after it is fully available.

intermediate

What programming model does Dataflow use to build pipelines?

Dataflow uses the Apache Beam programming model, which lets you write your data processing logic once and run it in either batch or streaming mode.

intermediate

How does Dataflow handle windowing in stream processing?

Dataflow groups streaming data into windows based on time or other criteria, so you can process data in manageable chunks even though it arrives continuously.

beginner

Why is Dataflow considered serverless?

Dataflow is serverless because Google manages the infrastructure, scaling, and resource allocation automatically, so you only focus on your data processing logic without managing servers.

Which of the following best describes batch processing in Dataflow?

A Processing data continuously as it arrives

B Processing data in fixed-size chunks after collection

C Processing only real-time data streams

D Processing data without any delay

Batch processing handles data in fixed-size chunks or groups after the data is collected, unlike stream processing which handles data continuously.

What programming model does Dataflow use to support both batch and streaming pipelines?

A MapReduce

B Spark

C Hadoop

D Apache Beam

Dataflow uses Apache Beam, which allows writing pipelines that can run in batch or streaming mode.

In stream processing, what is the purpose of windowing?

A To group data into manageable time-based chunks

B To delete old data automatically

C To speed up batch jobs

D To store data permanently

Windowing groups continuous streaming data into time-based or other logical chunks for easier processing.

Which feature makes Dataflow serverless?

A Manual resource allocation

B You must manage virtual machines

C Automatic scaling and infrastructure management by Google

D Requires installing software on your servers

Dataflow is serverless because Google automatically manages scaling and infrastructure, so you don't manage servers.

Which type of Dataflow job would you use to process live sensor data continuously?

A Stream job

B Offline job

C Batch job

D Manual job

Stream jobs process data continuously as it arrives, ideal for live sensor data.

Explain how Google Cloud Dataflow supports both stream and batch processing in a single pipeline.

Think about how one code can work for both live and stored data.

Describe the benefits of using a serverless service like Dataflow for data processing.

Consider what you don't have to do when using serverless.