Data pipeline patterns in GCP - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is a data pipeline in cloud computing?

A data pipeline is a set of steps that move and process data from one place to another, often automatically and continuously, like a water pipe moving water from a source to a destination.

beginner

What is the 'batch processing' pattern in data pipelines?

Batch processing collects data over time and processes it all at once, like baking a big batch of cookies instead of one at a time.

beginner

Explain the 'streaming processing' pattern in data pipelines.

Streaming processing handles data continuously as it arrives, like a conveyor belt where items are processed one by one without waiting.

intermediate

What is the role of Google Cloud Pub/Sub in data pipelines?

Google Cloud Pub/Sub acts like a message bus that moves data messages between services quickly and reliably, supporting streaming data pipelines.

intermediate

Why use Dataflow in a data pipeline?

Dataflow runs data processing tasks that can handle both batch and streaming data, automatically scaling and managing resources so you don't have to worry about the details.

Which data pipeline pattern processes data in real-time as it arrives?

A Streaming processing

B Batch processing

C Manual processing

D Archival processing

Streaming processing handles data continuously as it arrives, enabling real-time data handling.

What GCP service is best for messaging between components in a streaming pipeline?

A Cloud Storage

B BigQuery

C Cloud Pub/Sub

D Cloud SQL

Cloud Pub/Sub is designed for messaging and event delivery in streaming pipelines.

Which service can handle both batch and streaming data processing in GCP?

A Dataflow

B Cloud Functions

C Cloud Run

D Cloud Spanner

Dataflow supports unified batch and streaming data processing.

Batch processing is best described as:

A Processing data one piece at a time immediately

B Ignoring data until manually triggered

C Processing data only during business hours

D Collecting data and processing it all at once

Batch processing collects data over time and processes it together.

Which pattern would you choose for a pipeline that needs to analyze sensor data instantly?

A Batch processing

B Streaming processing

C Archival processing

D Manual processing

Streaming processing is suitable for instant analysis of continuous data like sensor data.

Describe the main differences between batch and streaming data pipeline patterns.

Think about how and when data is processed.

Explain how Google Cloud services like Pub/Sub and Dataflow work together in a streaming data pipeline.

Consider messaging and processing roles.