Data pipeline patterns in GCP - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Architecture

intermediate

Identify the correct data pipeline pattern for batch processing

You want to process large amounts of data collected over a day and run analytics once daily. Which data pipeline pattern fits best?

A Streaming pipeline that processes data in real-time as it arrives.

B Lambda architecture combining batch and streaming for fault tolerance.

C Batch pipeline that processes data in scheduled chunks, like daily batches.

D Micro-batch pipeline that processes data every few seconds.

💡 Hint

Think about when the data is processed: all at once or continuously.

Batch pipelines process data in large chunks at scheduled times, ideal for daily analytics.

❓ service_behavior

intermediate

What happens when a streaming pipeline in GCP Pub/Sub loses a message?

In a streaming pipeline using Google Cloud Pub/Sub, if a message is not acknowledged by the subscriber, what is the expected behavior?

A The message is deleted immediately and lost permanently.

B The message is retried and redelivered until acknowledged or expires.

C The message is sent to a dead-letter queue instantly.

D The message is duplicated and sent to all subscribers.

💡 Hint

Consider how Pub/Sub ensures message delivery reliability.

Pub/Sub retries delivery of unacknowledged messages until they are acknowledged or expire.

❓ security

advanced

Secure data transfer in a multi-stage pipeline

You have a multi-stage data pipeline in GCP involving Cloud Storage, Dataflow, and BigQuery. Which practice best secures data in transit between these services?

A Enable VPC Service Controls and use private IPs for communication.

B Use FTP protocol for faster data transfer.

C Encrypt data only at rest, not during transfer.

D Use public internet with IP whitelisting for data transfer.

💡 Hint

Think about how to keep data inside Google's private network.

VPC Service Controls and private IPs keep data transfers within Google's secure network, protecting data in transit.

🧠 Conceptual

advanced

Choosing the right pipeline pattern for low-latency analytics

You need to analyze data with minimal delay after it is generated. Which data pipeline pattern is most suitable?

A Streaming processing with event-driven triggers.

B Batch processing with daily scheduled jobs.

C Micro-batch processing with hourly intervals.

D Manual data export and import for analysis.

💡 Hint

Consider how quickly data should be available for analysis.

Streaming pipelines process data as it arrives, enabling low-latency analytics.

✅ Best Practice

expert

Optimizing cost and performance in a hybrid data pipeline

You have a hybrid data pipeline combining batch and streaming in GCP. What is the best practice to optimize cost without sacrificing performance?

A Run streaming jobs continuously on large instances regardless of load.

B Use only batch processing to avoid streaming costs.

C Schedule batch jobs during peak hours and disable autoscaling.

D Use autoscaling for streaming jobs and schedule batch jobs during off-peak hours.

💡 Hint

Think about adjusting resources based on workload and timing.

Autoscaling streaming jobs adjusts resources to demand, and scheduling batch jobs off-peak reduces costs while maintaining performance.