Dataflow for stream/batch processing in GCP - Step-by-Step Execution

Process Flow - Dataflow for stream/batch processing

Input Data Source

↓

Dataflow Job Start

↓

Choose Processing Mode

↓

Batch

↓

Read All Data

↓

Apply Transformations

↓

Write Output Data

↓

Job Complete or Continues

Dataflow reads data from a source, then processes it either as a batch (all at once) or stream (continuously), applies transformations, and writes results.

Execution Sample

GCP

pipeline = DataflowPipeline()
if mode == 'batch':
  data = pipeline.read_all(source)
else:
  data = pipeline.read_stream(source)
result = data.apply_transformations()
pipeline.write(result, sink)

This code sets up a Dataflow pipeline that reads data in batch or stream mode, processes it, and writes the output.

Process Table

Step	Action	Mode	Data State	Output/Result
1	Start Dataflow job	batch or stream	No data read yet	Job initialized
2	Check mode	batch	No data read yet	Batch mode selected
3	Read all data from source	batch	All data loaded	Data ready for processing
4	Apply transformations	batch	Data loaded	Transformed data ready
5	Write output data	batch	Transformed data	Data written to sink
6	Job completes	batch	Output written	Job finished successfully
1	Start Dataflow job	stream	No data read yet	Job initialized
2	Check mode	stream	No data read yet	Stream mode selected
3	Read data continuously	stream	Data arriving continuously	Data stream open
4	Apply transformations continuously	stream	Data streaming	Transformed data streaming
5	Write output data continuously	stream	Transformed data streaming	Output stream updated
6	Job runs continuously	stream	Continuous processing	Job runs until stopped

💡 Batch job ends after processing all data; stream job runs continuously until stopped.

Status Tracker

Variable	Start	After Step 3 (Batch)	After Step 4 (Batch)	After Step 5 (Batch)	After Step 3 (Stream)	After Step 4 (Stream)	After Step 5 (Stream)
mode	undefined	batch	batch	batch	stream	stream	stream
data	empty	all data loaded	transformed data	transformed data	stream open	transformed stream	transformed stream
job_state	initialized	running	running	completed	running	running	running

Key Moments - 3 Insights

Why does the batch job stop after step 6 but the stream job keeps running?

Because batch mode processes all data at once and then finishes (see execution_table row 6), while stream mode processes data continuously and runs until manually stopped (row 12).

What changes in data reading between batch and stream modes?

Batch mode reads all data at once (row 3), while stream mode reads data continuously as it arrives (row 9). This affects how transformations and output writing happen.

How does the variable 'data' differ after reading in batch vs stream?

In batch, 'data' holds all loaded data at once (variable_tracker after step 3 batch), but in stream, 'data' represents an ongoing stream of data (variable_tracker after step 3 stream).

Visual Quiz - 3 Questions

Look at the execution_table, at which step does the batch job write output data?

A Step 4

B Step 6

C Step 5

D Step 3

💡 Hint

Check the 'Action' and 'Output/Result' columns for batch mode in execution_table rows 4-6.

According to variable_tracker, what is the state of 'job_state' after step 5 in batch mode?

A completed

B running

C initialized

D stopped

💡 Hint

Look at the 'job_state' row under 'After Step 5 (Batch)' in variable_tracker.

If the mode changes from 'batch' to 'stream', how does the data reading step change in execution_table?

A Reads all data at once

B Reads data continuously

C Does not read data

D Reads data after writing output

💡 Hint

Compare execution_table rows 3 (batch) and 9 (stream) under 'Action' and 'Data State'.

Concept Snapshot

Dataflow processes data in two modes:
- Batch: reads all data, processes, then finishes.
- Stream: reads data continuously, processes continuously.
Choose mode at job start.
Apply transformations to data.
Write results to output.
Batch jobs end; stream jobs run until stopped.

Full Transcript

Dataflow is a Google Cloud service that processes data either in batch or streaming mode. The job starts by reading data from a source. In batch mode, it reads all data at once, applies transformations, writes the output, and then finishes. In streaming mode, it reads data continuously as it arrives, applies transformations continuously, writes output continuously, and runs until manually stopped. Variables like 'data' and 'job_state' change differently depending on the mode. Batch jobs have a clear end, while streaming jobs run indefinitely. Understanding these steps helps in designing data processing pipelines effectively.

Practice

1. What is the main difference between streaming and batch processing in Google Cloud Dataflow?

easy

A. Batch processing is faster than streaming in all cases.

B. Streaming processes data continuously as it arrives; batch processes fixed sets of data.

C. Streaming only reads data; batch only writes data.

D. Batch processes data continuously; streaming processes fixed sets of data.

Solution

1. Step 1: Understand streaming processing

   Streaming processes data continuously as new data arrives in real time.

2. Step 2: Understand batch processing

   Batch processes data in fixed sets or chunks, not continuously.

3. Final Answer:

   Streaming processes data continuously as it arrives; batch processes fixed sets of data. -> Option B

4. Quick Check:

   Streaming = continuous, Batch = fixed sets [OK]

Hint: Streaming = live data; batch = fixed data sets [OK]

Common Mistakes:

• Confusing batch as continuous processing
• Thinking streaming only reads data
• Assuming batch is always faster

2. Which option correctly sets a Dataflow pipeline to run in streaming mode using the Python SDK?

easy

A. options.view_as(StandardOptions).batch = True

B. options.view_as(StandardOptions).stream = False

C. options.view_as(StandardOptions).mode = 'batch'

D. options.view_as(StandardOptions).streaming = True

Solution

1. Step 1: Identify the correct property for streaming

   In Dataflow Python SDK, streaming mode is enabled by setting streaming = True on StandardOptions.

2. Step 2: Check other options

   Batch is default and does not require setting batch = True; mode property does not exist; stream = False disables streaming.

3. Final Answer:

   options.view_as(StandardOptions).streaming = True -> Option D

4. Quick Check:

   streaming = True enables streaming mode [OK]

Hint: Set streaming=True in StandardOptions for streaming mode [OK]

Common Mistakes:

• Using batch=True which is invalid
• Setting mode='batch' which is not a valid property
• Confusing stream with streaming property

3. Given this Python Dataflow pipeline snippet, what will be the output if run in batch mode on input [1, 2, 3]?

import apache_beam as beam
with beam.Pipeline() as p:
  result = (
    p | beam.Create([1, 2, 3])
      | beam.Map(lambda x: x * 2)
      | beam.Map(lambda x: x + 1)
  )
  result | beam.Map(print)

medium

A. 3, 5, 7

B. 1, 2, 3

C. 2, 4, 6

D. Error due to missing streaming option

Solution

1. Step 1: Trace the transformations

   Input list is [1, 2, 3]. First Map multiplies each by 2: [2, 4, 6]. Second Map adds 1 to each: [3, 5, 7].

2. Step 2: Confirm batch mode output

   Batch mode processes all data at once, so output will be printed as 3, 5, 7.

3. Final Answer:

   3, 5, 7 -> Option A

4. Quick Check:

   Multiply by 2 then add 1 = 3,5,7 [OK]

Hint: Apply transformations stepwise to input list [OK]

Common Mistakes:

• Forgetting to multiply before adding
• Assuming streaming mode affects output here
• Expecting original input as output

4. You wrote a Dataflow streaming pipeline but it fails to process live data. Which fix is most likely correct?

medium

A. Set options.view_as(StandardOptions).streaming = True before running the pipeline.

B. Change the input source to a batch file instead of a streaming source.

C. Remove all Map transforms to simplify the pipeline.

D. Run the pipeline without specifying any options.

Solution

1. Step 1: Identify streaming mode requirement

   Streaming pipelines require setting streaming=True in options to process live data.

2. Step 2: Evaluate other options

   Changing input to batch file disables streaming; removing transforms breaks logic; no options means default batch mode.

3. Final Answer:

   Set options.view_as(StandardOptions).streaming = True before running the pipeline. -> Option A

4. Quick Check:

   Enable streaming option to fix live data processing [OK]

Hint: Enable streaming=True to fix live data issues [OK]

Common Mistakes:

• Not enabling streaming mode in options
• Switching to batch input source mistakenly
• Removing necessary transforms

5. You want to build a Dataflow pipeline that reads live sensor data, filters out values below 50, and writes results to BigQuery. Which approach is best?

hard

A. Use streaming mode with a Cloud Storage file source, apply a Map transform, then write to BigQuery sink.

B. Use batch mode with a Cloud Storage file source, apply a Map transform, then write to BigQuery sink.

C. Use streaming mode with a Pub/Sub source, apply a Filter transform, then write to BigQuery sink.

D. Use batch mode with a Pub/Sub source, apply a Filter transform, then write to BigQuery sink.

Solution

1. Step 1: Choose streaming mode for live sensor data

   Live sensor data requires streaming mode to process data as it arrives.

2. Step 2: Select Pub/Sub as source and apply Filter transform

   Pub/Sub is the streaming data source; Filter removes values below 50; BigQuery sink stores results.

3. Final Answer:

   Use streaming mode with a Pub/Sub source, apply a Filter transform, then write to BigQuery sink. -> Option C

4. Quick Check:

   Streaming + Pub/Sub + Filter + BigQuery = correct pipeline [OK]

Hint: Live data needs streaming + Pub/Sub + Filter + BigQuery [OK]

Common Mistakes:

• Using batch mode for live data
• Using Cloud Storage as streaming source
• Applying Map instead of Filter for filtering