Data pipeline patterns in GCP - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Architecture

intermediate

Identify the correct data pipeline pattern for batch processing

You want to process large amounts of data collected over a day and run analytics once daily. Which data pipeline pattern fits best?

A Streaming pipeline that processes data in real-time as it arrives.

B Lambda architecture combining batch and streaming for fault tolerance.

C Batch pipeline that processes data in scheduled chunks, like daily batches.

D Micro-batch pipeline that processes data every few seconds.

💡 Hint

Think about when the data is processed: all at once or continuously.

Batch pipelines process data in large chunks at scheduled times, ideal for daily analytics.

❓ service_behavior

intermediate

What happens when a streaming pipeline in GCP Pub/Sub loses a message?

In a streaming pipeline using Google Cloud Pub/Sub, if a message is not acknowledged by the subscriber, what is the expected behavior?

A The message is deleted immediately and lost permanently.

B The message is retried and redelivered until acknowledged or expires.

C The message is sent to a dead-letter queue instantly.

D The message is duplicated and sent to all subscribers.

💡 Hint

Consider how Pub/Sub ensures message delivery reliability.

Pub/Sub retries delivery of unacknowledged messages until they are acknowledged or expire.

❓ security

advanced

Secure data transfer in a multi-stage pipeline

You have a multi-stage data pipeline in GCP involving Cloud Storage, Dataflow, and BigQuery. Which practice best secures data in transit between these services?

A Enable VPC Service Controls and use private IPs for communication.

B Use FTP protocol for faster data transfer.

C Encrypt data only at rest, not during transfer.

D Use public internet with IP whitelisting for data transfer.

💡 Hint

Think about how to keep data inside Google's private network.

VPC Service Controls and private IPs keep data transfers within Google's secure network, protecting data in transit.

🧠 Conceptual

advanced

Choosing the right pipeline pattern for low-latency analytics

You need to analyze data with minimal delay after it is generated. Which data pipeline pattern is most suitable?

A Streaming processing with event-driven triggers.

B Batch processing with daily scheduled jobs.

C Micro-batch processing with hourly intervals.

D Manual data export and import for analysis.

💡 Hint

Consider how quickly data should be available for analysis.

Streaming pipelines process data as it arrives, enabling low-latency analytics.

✅ Best Practice

expert

Optimizing cost and performance in a hybrid data pipeline

You have a hybrid data pipeline combining batch and streaming in GCP. What is the best practice to optimize cost without sacrificing performance?

A Run streaming jobs continuously on large instances regardless of load.

B Use only batch processing to avoid streaming costs.

C Schedule batch jobs during peak hours and disable autoscaling.

D Use autoscaling for streaming jobs and schedule batch jobs during off-peak hours.

💡 Hint

Think about adjusting resources based on workload and timing.

Autoscaling streaming jobs adjusts resources to demand, and scheduling batch jobs off-peak reduces costs while maintaining performance.

Practice

1. Which Google Cloud service is best suited for real-time messaging in a data pipeline?

easy

A. Cloud Storage

B. BigQuery

C. Pub/Sub

D. Cloud Functions

Solution

1. Step 1: Understand real-time messaging needs

   Real-time messaging requires a service that can quickly send and receive messages between components.

2. Step 2: Identify Google Cloud services for messaging

   Pub/Sub is designed for real-time messaging, while BigQuery is for analytics, Cloud Storage for file storage, and Cloud Functions for event-driven compute.

3. Final Answer:

   Pub/Sub -> Option C

4. Quick Check:

   Real-time messaging = Pub/Sub [OK]

Hint: Real-time messaging in GCP? Think Pub/Sub first [OK]

Common Mistakes:

• Confusing BigQuery as a messaging service
• Choosing Cloud Storage for streaming data
• Selecting Cloud Functions instead of messaging service

2. Which of the following is the correct syntax to create a Dataflow pipeline in Python using Apache Beam?

easy

A. pipeline = beam.Pipeline(options=options)

B. pipeline = Dataflow(options)

C. pipeline = beam.Create(options)

D. pipeline = DataflowPipeline(options=options)

Solution

1. Step 1: Recall Apache Beam pipeline creation syntax

   Apache Beam pipelines are created using beam.Pipeline with options passed as a parameter.

2. Step 2: Compare options

   pipeline = beam.Pipeline(options=options) matches the correct syntax. Other options use incorrect class names or methods.

3. Final Answer:

   pipeline = beam.Pipeline(options=options) -> Option A

4. Quick Check:

   Apache Beam pipeline creation = beam.Pipeline [OK]

Hint: Beam pipelines start with beam.Pipeline(options=...) [OK]

Common Mistakes:

• Using Dataflow instead of beam.Pipeline
• Confusing Create with pipeline creation
• Incorrect class names like DataflowPipeline

3. Given this simplified Dataflow pipeline code snippet, what will be the output in BigQuery?

p = beam.Pipeline(options=options)
(p
 | 'Read' >> beam.io.ReadFromPubSub(subscription='sub')
 | 'Parse' >> beam.Map(lambda x: x.decode('utf-8'))
 | 'Write' >> beam.io.WriteToBigQuery('project:dataset.table'))
p.run()

medium

A. Data will be stored in Cloud Storage instead of BigQuery

B. Data from Pub/Sub messages decoded and stored in BigQuery table

C. Pipeline will fail due to missing schema in WriteToBigQuery

D. Raw Pub/Sub messages stored as bytes in BigQuery

Solution

1. Step 1: Analyze pipeline steps

   The pipeline reads messages from Pub/Sub, decodes bytes to strings, then writes to BigQuery.

2. Step 2: Understand output behavior

   Since messages are decoded before writing, the data stored in BigQuery will be the decoded string form.

3. Final Answer:

   Data from Pub/Sub messages decoded and stored in BigQuery table -> Option B

4. Quick Check:

   Decoded Pub/Sub data written to BigQuery = Data from Pub/Sub messages decoded and stored in BigQuery table [OK]

Hint: Decoded Pub/Sub data goes to BigQuery if WriteToBigQuery used [OK]

Common Mistakes:

• Assuming raw bytes are stored without decoding
• Thinking pipeline fails without explicit schema (default schema can be inferred)
• Confusing output destination as Cloud Storage

4. You have a Dataflow pipeline that reads from Pub/Sub but fails with a permission error. What is the most likely cause?

medium

A. The Pub/Sub topic does not exist

B. BigQuery dataset is not created

C. The Dataflow pipeline code has syntax errors

D. The service account running Dataflow lacks Pub/Sub Subscriber role

Solution

1. Step 1: Identify permission error context

   Permission errors usually mean the service account lacks rights to access a resource.

2. Step 2: Match error to resource

   Since the pipeline reads from Pub/Sub, the service account needs the Subscriber role on the Pub/Sub topic.

3. Final Answer:

   The service account running Dataflow lacks Pub/Sub Subscriber role -> Option D

4. Quick Check:

   Permission error reading Pub/Sub = missing Subscriber role [OK]

Hint: Permission errors often mean missing roles on resources [OK]

Common Mistakes:

• Assuming syntax errors cause permission errors
• Thinking topic absence causes permission errors
• Confusing BigQuery dataset issues with Pub/Sub permissions

5. You want to build a data pipeline that ingests streaming IoT sensor data, processes it in real-time, and stores aggregated results hourly in BigQuery. Which pattern best fits this use case?

hard

A. Use Pub/Sub for ingestion, Dataflow streaming for processing with windowing, then write to BigQuery

B. Use Cloud Storage for ingestion, batch Dataflow jobs for processing, then write to BigQuery

C. Use BigQuery streaming inserts directly from IoT devices

D. Use Cloud Functions triggered by Cloud Storage uploads to process data

Solution

1. Step 1: Identify streaming ingestion and processing needs

   IoT sensor data is streaming and requires real-time processing with time-based aggregation.

2. Step 2: Match GCP services to requirements

   Pub/Sub handles streaming ingestion, Dataflow streaming supports windowed aggregation, and BigQuery stores results efficiently.

3. Final Answer:

   Use Pub/Sub for ingestion, Dataflow streaming for processing with windowing, then write to BigQuery -> Option A

4. Quick Check:

   Streaming ingestion + windowed processing + BigQuery = Use Pub/Sub for ingestion, Dataflow streaming for processing with windowing, then write to BigQuery [OK]

Hint: Streaming + windowed aggregation = Pub/Sub + Dataflow + BigQuery [OK]

Common Mistakes:

• Using batch processing for streaming data
• Trying to write streaming data directly to BigQuery without processing
• Using Cloud Storage which is for batch files, not streaming