Dataflow for stream/batch processing in GCP - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is Google Cloud Dataflow?

Google Cloud Dataflow is a fully managed service for processing and analyzing large data sets in real time or batch mode. It helps you build data pipelines that can handle streaming or batch data easily.

beginner

What is the difference between stream processing and batch processing in Dataflow?

Stream processing handles data continuously as it arrives, like watching a live video. Batch processing handles data in chunks or groups, like watching a recorded video after it is fully available.

intermediate

What programming model does Dataflow use to build pipelines?

Dataflow uses the Apache Beam programming model, which lets you write your data processing logic once and run it in either batch or streaming mode.

intermediate

How does Dataflow handle windowing in stream processing?

Dataflow groups streaming data into windows based on time or other criteria, so you can process data in manageable chunks even though it arrives continuously.

beginner

Why is Dataflow considered serverless?

Dataflow is serverless because Google manages the infrastructure, scaling, and resource allocation automatically, so you only focus on your data processing logic without managing servers.

Which of the following best describes batch processing in Dataflow?

A Processing data continuously as it arrives

B Processing data in fixed-size chunks after collection

C Processing only real-time data streams

D Processing data without any delay

Batch processing handles data in fixed-size chunks or groups after the data is collected, unlike stream processing which handles data continuously.

What programming model does Dataflow use to support both batch and streaming pipelines?

A MapReduce

B Spark

C Hadoop

D Apache Beam

Dataflow uses Apache Beam, which allows writing pipelines that can run in batch or streaming mode.

In stream processing, what is the purpose of windowing?

A To group data into manageable time-based chunks

B To delete old data automatically

C To speed up batch jobs

D To store data permanently

Windowing groups continuous streaming data into time-based or other logical chunks for easier processing.

Which feature makes Dataflow serverless?

A Manual resource allocation

B You must manage virtual machines

C Automatic scaling and infrastructure management by Google

D Requires installing software on your servers

Dataflow is serverless because Google automatically manages scaling and infrastructure, so you don't manage servers.

Which type of Dataflow job would you use to process live sensor data continuously?

A Stream job

B Offline job

C Batch job

D Manual job

Stream jobs process data continuously as it arrives, ideal for live sensor data.

Explain how Google Cloud Dataflow supports both stream and batch processing in a single pipeline.

Think about how one code can work for both live and stored data.

Describe the benefits of using a serverless service like Dataflow for data processing.

Consider what you don't have to do when using serverless.

Practice

1. What is the main difference between streaming and batch processing in Google Cloud Dataflow?

easy

A. Batch processing is faster than streaming in all cases.

B. Streaming processes data continuously as it arrives; batch processes fixed sets of data.

C. Streaming only reads data; batch only writes data.

D. Batch processes data continuously; streaming processes fixed sets of data.

Solution

1. Step 1: Understand streaming processing

   Streaming processes data continuously as new data arrives in real time.

2. Step 2: Understand batch processing

   Batch processes data in fixed sets or chunks, not continuously.

3. Final Answer:

   Streaming processes data continuously as it arrives; batch processes fixed sets of data. -> Option B

4. Quick Check:

   Streaming = continuous, Batch = fixed sets [OK]

Hint: Streaming = live data; batch = fixed data sets [OK]

Common Mistakes:

• Confusing batch as continuous processing
• Thinking streaming only reads data
• Assuming batch is always faster

2. Which option correctly sets a Dataflow pipeline to run in streaming mode using the Python SDK?

easy

A. options.view_as(StandardOptions).batch = True

B. options.view_as(StandardOptions).stream = False

C. options.view_as(StandardOptions).mode = 'batch'

D. options.view_as(StandardOptions).streaming = True

Solution

1. Step 1: Identify the correct property for streaming

   In Dataflow Python SDK, streaming mode is enabled by setting streaming = True on StandardOptions.

2. Step 2: Check other options

   Batch is default and does not require setting batch = True; mode property does not exist; stream = False disables streaming.

3. Final Answer:

   options.view_as(StandardOptions).streaming = True -> Option D

4. Quick Check:

   streaming = True enables streaming mode [OK]

Hint: Set streaming=True in StandardOptions for streaming mode [OK]

Common Mistakes:

• Using batch=True which is invalid
• Setting mode='batch' which is not a valid property
• Confusing stream with streaming property

3. Given this Python Dataflow pipeline snippet, what will be the output if run in batch mode on input [1, 2, 3]?

import apache_beam as beam
with beam.Pipeline() as p:
  result = (
    p | beam.Create([1, 2, 3])
      | beam.Map(lambda x: x * 2)
      | beam.Map(lambda x: x + 1)
  )
  result | beam.Map(print)

medium

A. 3, 5, 7

B. 1, 2, 3

C. 2, 4, 6

D. Error due to missing streaming option

Solution

1. Step 1: Trace the transformations

   Input list is [1, 2, 3]. First Map multiplies each by 2: [2, 4, 6]. Second Map adds 1 to each: [3, 5, 7].

2. Step 2: Confirm batch mode output

   Batch mode processes all data at once, so output will be printed as 3, 5, 7.

3. Final Answer:

   3, 5, 7 -> Option A

4. Quick Check:

   Multiply by 2 then add 1 = 3,5,7 [OK]

Hint: Apply transformations stepwise to input list [OK]

Common Mistakes:

• Forgetting to multiply before adding
• Assuming streaming mode affects output here
• Expecting original input as output

4. You wrote a Dataflow streaming pipeline but it fails to process live data. Which fix is most likely correct?

medium

A. Set options.view_as(StandardOptions).streaming = True before running the pipeline.

B. Change the input source to a batch file instead of a streaming source.

C. Remove all Map transforms to simplify the pipeline.

D. Run the pipeline without specifying any options.

Solution

1. Step 1: Identify streaming mode requirement

   Streaming pipelines require setting streaming=True in options to process live data.

2. Step 2: Evaluate other options

   Changing input to batch file disables streaming; removing transforms breaks logic; no options means default batch mode.

3. Final Answer:

   Set options.view_as(StandardOptions).streaming = True before running the pipeline. -> Option A

4. Quick Check:

   Enable streaming option to fix live data processing [OK]

Hint: Enable streaming=True to fix live data issues [OK]

Common Mistakes:

• Not enabling streaming mode in options
• Switching to batch input source mistakenly
• Removing necessary transforms

5. You want to build a Dataflow pipeline that reads live sensor data, filters out values below 50, and writes results to BigQuery. Which approach is best?

hard

A. Use streaming mode with a Cloud Storage file source, apply a Map transform, then write to BigQuery sink.

B. Use batch mode with a Cloud Storage file source, apply a Map transform, then write to BigQuery sink.

C. Use streaming mode with a Pub/Sub source, apply a Filter transform, then write to BigQuery sink.

D. Use batch mode with a Pub/Sub source, apply a Filter transform, then write to BigQuery sink.

Solution

1. Step 1: Choose streaming mode for live sensor data

   Live sensor data requires streaming mode to process data as it arrives.

2. Step 2: Select Pub/Sub as source and apply Filter transform

   Pub/Sub is the streaming data source; Filter removes values below 50; BigQuery sink stores results.

3. Final Answer:

   Use streaming mode with a Pub/Sub source, apply a Filter transform, then write to BigQuery sink. -> Option C

4. Quick Check:

   Streaming + Pub/Sub + Filter + BigQuery = correct pipeline [OK]

Hint: Live data needs streaming + Pub/Sub + Filter + BigQuery [OK]

Common Mistakes:

• Using batch mode for live data
• Using Cloud Storage as streaming source
• Applying Map instead of Filter for filtering