Dataflow for stream/batch processing in GCP - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ service_behavior

intermediate

Understanding Dataflow Windowing Behavior

You have a Dataflow pipeline processing streaming data with fixed windows of 5 minutes. What will happen if late data arrives after the window has closed and the allowed lateness period has passed?

A The late data will be discarded and not processed in the pipeline.

B The pipeline will reprocess the entire window including the late data.

C The late data will be processed immediately in a new window.

D The pipeline will throw an error and stop processing.

💡 Hint

Think about how Dataflow handles data that arrives after the window and allowed lateness period.

Dataflow discards late data that arrives after the window and allowed lateness period to maintain consistent results and avoid reprocessing.

❓ Architecture

intermediate

Choosing Batch vs Streaming in Dataflow

You need to process a large dataset that updates once a day and produce a report. Which Dataflow processing mode is most appropriate?

A Batch mode, to process the entire dataset once daily.

B Streaming mode, to process data as it arrives continuously.

C Streaming mode with fixed windows of 1 day.

D Batch mode with micro-batches every few minutes.

💡 Hint

Consider how often the data updates and the best way to process large static datasets.

Batch mode is best for large datasets that update periodically, as it processes the entire dataset efficiently once per update.

❓ security

advanced

Securing Dataflow Pipeline Access

You want to restrict who can start and manage your Dataflow jobs in your GCP project. Which IAM role should you assign to users to allow them to create and cancel Dataflow jobs but not modify other resources?

A roles/viewer

B roles/dataflow.admin

C roles/dataflow.worker

D roles/dataflow.developer

💡 Hint

Think about the role that allows job management but limits broader admin permissions.

The Dataflow Developer role allows users to create and cancel jobs without full admin rights.

❓ Configuration

advanced

Configuring Autoscaling in Dataflow

You want your Dataflow streaming job to automatically adjust the number of worker instances based on workload. Which autoscaling algorithm should you choose for best responsiveness?

A CPU_BASED - scale workers based on CPU usage.

B NONE - disable autoscaling and set fixed workers.

C THROUGHPUT_BASED - scale workers based on data throughput.

D MEMORY_BASED - scale workers based on memory usage.

💡 Hint

Consider which metric best reflects workload changes in streaming data.

THROUGHPUT_BASED autoscaling adjusts workers based on data volume, providing responsive scaling for streaming jobs.

✅ Best Practice

expert

Optimizing Dataflow Pipeline for Cost and Performance

You have a Dataflow batch pipeline that processes large files daily. You want to reduce cost without significantly increasing processing time. Which combination of strategies is best?

A Use autoscaling with minimum workers set to zero and disable streaming engine.

B Use autoscaling with a maximum number of workers and enable shuffle optimization.

C Use the smallest machine type and disable shuffle optimization to save cost.

D Disable autoscaling and set a high fixed number of workers to finish faster.

💡 Hint

Think about balancing worker count and pipeline optimizations.

Autoscaling with a max worker limit controls cost, and shuffle optimization improves performance by reducing data movement.

Practice

1. What is the main difference between streaming and batch processing in Google Cloud Dataflow?

easy

A. Batch processing is faster than streaming in all cases.

B. Streaming processes data continuously as it arrives; batch processes fixed sets of data.

C. Streaming only reads data; batch only writes data.

D. Batch processes data continuously; streaming processes fixed sets of data.

Solution

1. Step 1: Understand streaming processing

   Streaming processes data continuously as new data arrives in real time.

2. Step 2: Understand batch processing

   Batch processes data in fixed sets or chunks, not continuously.

3. Final Answer:

   Streaming processes data continuously as it arrives; batch processes fixed sets of data. -> Option B

4. Quick Check:

   Streaming = continuous, Batch = fixed sets [OK]

Hint: Streaming = live data; batch = fixed data sets [OK]

Common Mistakes:

• Confusing batch as continuous processing
• Thinking streaming only reads data
• Assuming batch is always faster

2. Which option correctly sets a Dataflow pipeline to run in streaming mode using the Python SDK?

easy

A. options.view_as(StandardOptions).batch = True

B. options.view_as(StandardOptions).stream = False

C. options.view_as(StandardOptions).mode = 'batch'

D. options.view_as(StandardOptions).streaming = True

Solution

1. Step 1: Identify the correct property for streaming

   In Dataflow Python SDK, streaming mode is enabled by setting streaming = True on StandardOptions.

2. Step 2: Check other options

   Batch is default and does not require setting batch = True; mode property does not exist; stream = False disables streaming.

3. Final Answer:

   options.view_as(StandardOptions).streaming = True -> Option D

4. Quick Check:

   streaming = True enables streaming mode [OK]

Hint: Set streaming=True in StandardOptions for streaming mode [OK]

Common Mistakes:

• Using batch=True which is invalid
• Setting mode='batch' which is not a valid property
• Confusing stream with streaming property

3. Given this Python Dataflow pipeline snippet, what will be the output if run in batch mode on input [1, 2, 3]?

import apache_beam as beam
with beam.Pipeline() as p:
  result = (
    p | beam.Create([1, 2, 3])
      | beam.Map(lambda x: x * 2)
      | beam.Map(lambda x: x + 1)
  )
  result | beam.Map(print)

medium

A. 3, 5, 7

B. 1, 2, 3

C. 2, 4, 6

D. Error due to missing streaming option

Solution

1. Step 1: Trace the transformations

   Input list is [1, 2, 3]. First Map multiplies each by 2: [2, 4, 6]. Second Map adds 1 to each: [3, 5, 7].

2. Step 2: Confirm batch mode output

   Batch mode processes all data at once, so output will be printed as 3, 5, 7.

3. Final Answer:

   3, 5, 7 -> Option A

4. Quick Check:

   Multiply by 2 then add 1 = 3,5,7 [OK]

Hint: Apply transformations stepwise to input list [OK]

Common Mistakes:

• Forgetting to multiply before adding
• Assuming streaming mode affects output here
• Expecting original input as output

4. You wrote a Dataflow streaming pipeline but it fails to process live data. Which fix is most likely correct?

medium

A. Set options.view_as(StandardOptions).streaming = True before running the pipeline.

B. Change the input source to a batch file instead of a streaming source.

C. Remove all Map transforms to simplify the pipeline.

D. Run the pipeline without specifying any options.

Solution

1. Step 1: Identify streaming mode requirement

   Streaming pipelines require setting streaming=True in options to process live data.

2. Step 2: Evaluate other options

   Changing input to batch file disables streaming; removing transforms breaks logic; no options means default batch mode.

3. Final Answer:

   Set options.view_as(StandardOptions).streaming = True before running the pipeline. -> Option A

4. Quick Check:

   Enable streaming option to fix live data processing [OK]

Hint: Enable streaming=True to fix live data issues [OK]

Common Mistakes:

• Not enabling streaming mode in options
• Switching to batch input source mistakenly
• Removing necessary transforms

5. You want to build a Dataflow pipeline that reads live sensor data, filters out values below 50, and writes results to BigQuery. Which approach is best?

hard

A. Use streaming mode with a Cloud Storage file source, apply a Map transform, then write to BigQuery sink.

B. Use batch mode with a Cloud Storage file source, apply a Map transform, then write to BigQuery sink.

C. Use streaming mode with a Pub/Sub source, apply a Filter transform, then write to BigQuery sink.

D. Use batch mode with a Pub/Sub source, apply a Filter transform, then write to BigQuery sink.

Solution

1. Step 1: Choose streaming mode for live sensor data

   Live sensor data requires streaming mode to process data as it arrives.

2. Step 2: Select Pub/Sub as source and apply Filter transform

   Pub/Sub is the streaming data source; Filter removes values below 50; BigQuery sink stores results.

3. Final Answer:

   Use streaming mode with a Pub/Sub source, apply a Filter transform, then write to BigQuery sink. -> Option C

4. Quick Check:

   Streaming + Pub/Sub + Filter + BigQuery = correct pipeline [OK]

Hint: Live data needs streaming + Pub/Sub + Filter + BigQuery [OK]

Common Mistakes:

• Using batch mode for live data
• Using Cloud Storage as streaming source
• Applying Map instead of Filter for filtering