Dataflow for stream/batch processing in GCP - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to create a Dataflow pipeline that reads from a Pub/Sub topic.

GCP

pipeline = beam.Pipeline(options=PipelineOptions())
messages = pipeline | 'ReadFromPubSub' >> beam.io.ReadFromPubSub(topic=[1])

Drag options to blanks, or click blank then click option'

A"projects/my-project/topics/"

B"my-topic"

C"projects/my-project/subscriptions/my-subscription"

D"projects/my-project/topics/my-topic"

Attempts:

3 left

💡 Hint

Common Mistakes

Using only the topic name without the full path

Using subscription path instead of topic path

The Pub/Sub topic must be specified with the full resource path including project and topic name.

2fill in blank

medium

Complete the code to write the output of the pipeline to a BigQuery table.

GCP

output | 'WriteToBigQuery' >> beam.io.WriteToBigQuery(table=[1])

Drag options to blanks, or click blank then click option'

A"my-project:my_dataset.my_table"

B"my_dataset.my_table"

C"my_table"

D"my_dataset/my_table"

Attempts:

3 left

💡 Hint

Common Mistakes

Using only dataset.table without project

Using slashes instead of dots or vice versa

BigQuery table must be specified with the full resource path including project, dataset, and table.

3fill in blank

hard

Fix the error in the pipeline options to enable streaming mode.

GCP

options = PipelineOptions()
options.view_as(StandardOptions).[1] = True

Drag options to blanks, or click blank then click option'

Abatch_mode

Benable_streaming

Cstreaming

Dis_streaming

Attempts:

3 left

💡 Hint

Common Mistakes

Using incorrect attribute names like 'batch_mode' or 'enable_streaming'

Using 'is_streaming' which does not exist

The correct attribute to enable streaming mode is 'streaming' set to True.

4fill in blank

hard

Fill both blanks to create a windowed aggregation that sums values every 5 minutes.

GCP

windowed_data = data | 'Window' >> beam.WindowInto(beam.window.[1](5 * 60))
result = windowed_data | 'SumValues' >> beam.CombinePerKey([2])

Drag options to blanks, or click blank then click option'

AFixedWindows

BSlidingWindows

Csum

Dlen

Attempts:

3 left

💡 Hint

Common Mistakes

Using SlidingWindows instead of FixedWindows for fixed intervals

Using 'len' which counts elements instead of summing

FixedWindows creates fixed time windows. 'sum' sums values per key.

5fill in blank

hard

Fill both blanks to filter elements greater than 100 and map keys to uppercase.

GCP

filtered = data | 'FilterGreater' >> beam.Filter(lambda x: x[1] [1] 100)
mapped = filtered | 'MapKeys' >> beam.Map(lambda x: (x[0][2], x[1]))
result = mapped | 'GroupByKey' >> beam.GroupByKey()

Drag options to blanks, or click blank then click option'

A>

B.upper()

C<

D.lower()

Attempts:

3 left

💡 Hint

Common Mistakes

Using '<' instead of '>' in filter

Using '.lower()' instead of '.upper()' for keys

Filter uses '>' to keep values greater than 100. Map uses '.upper()' to uppercase keys. GroupByKey groups by keys.