Data pipeline patterns in GCP - Mini Project: Build & Apply

Data Pipeline Patterns on Google Cloud Platform

📖 Scenario: You work for a company that collects sales data from multiple stores. You want to build a simple data pipeline on Google Cloud Platform (GCP) to collect, process, and store this data efficiently.This pipeline will help the company analyze sales trends and make better decisions.

🎯 Goal: Build a basic data pipeline on GCP using Cloud Storage, Pub/Sub, Dataflow, and BigQuery.You will create the initial data source, configure a Pub/Sub topic, write a Dataflow pipeline to process messages, and set up a BigQuery table to store the results.

📋 What You'll Learn

Create a Cloud Storage bucket to hold raw sales data files

Create a Pub/Sub topic to receive messages about new data

Write a Dataflow pipeline that reads from Pub/Sub and writes to BigQuery

Create a BigQuery table to store processed sales data

💡 Why This Matters

🌍 Real World

Companies often collect data from many sources and need to process it in real-time or batch to gain insights. This project shows a simple way to build such a pipeline on GCP.

💼 Career

Understanding how to build data pipelines on cloud platforms like GCP is essential for roles in data engineering, cloud architecture, and analytics.

1

Create a Cloud Storage bucket for raw sales data

Create a Cloud Storage bucket named sales-raw-data in the us-central1 region to hold raw sales data files.

GCP

# Your code here to create the Cloud Storage bucket

Hint

Use the gcloud storage buckets create command with the --location flag.

2

Create a Pub/Sub topic for new data notifications

Create a Pub/Sub topic named sales-data-topic to receive messages when new sales data files arrive.

GCP

gcloud storage buckets create sales-raw-data --location=us-central1
# Your code here to create the Pub/Sub topic

Hint

Use the gcloud pubsub topics create command with the topic name.

3

Write a Dataflow pipeline to process Pub/Sub messages

Write a Dataflow pipeline in Python that reads messages from the Pub/Sub topic sales-data-topic and writes the processed data to a BigQuery table named sales_dataset.sales_table. Use the Apache Beam SDK and specify the project as my-gcp-project.

GCP

gcloud storage buckets create sales-raw-data --location=us-central1
gcloud pubsub topics create sales-data-topic
# Your code here to write the Dataflow pipeline

Hint

Use Apache Beam's ReadFromPubSub and WriteToBigQuery transforms with the correct topic and table names.

4

Create a BigQuery table to store processed sales data

Create a BigQuery dataset named sales_dataset and a table named sales_table with columns store_id (STRING), date (STRING), and sales (FLOAT) in the project my-gcp-project.

GCP

gcloud storage buckets create sales-raw-data --location=us-central1
gcloud pubsub topics create sales-data-topic
import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions

options = PipelineOptions(
    project='my-gcp-project',
    runner='DataflowRunner',
    temp_location='gs://sales-raw-data/temp'
)

with beam.Pipeline(options=options) as p:
    (p
     | 'ReadFromPubSub' >> beam.io.ReadFromPubSub(topic='projects/my-gcp-project/topics/sales-data-topic')
     | 'Decode' >> beam.Map(lambda x: x.decode('utf-8'))
     | 'ParseCSV' >> beam.Map(lambda line: dict(zip(['store_id', 'date', 'sales'], line.split(','))))
     | 'WriteToBigQuery' >> beam.io.WriteToBigQuery(
            'my-gcp-project:sales_dataset.sales_table',
            schema='store_id:STRING,date:STRING,sales:FLOAT',
            write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND
        )
    )
# Your code here to create BigQuery dataset and table

Hint

Use gcloud bigquery datasets create and gcloud bigquery tables create with the correct schema.

Practice

1. Which Google Cloud service is best suited for real-time messaging in a data pipeline?

easy

A. Cloud Storage

B. BigQuery

C. Pub/Sub

D. Cloud Functions

Solution

1. Step 1: Understand real-time messaging needs

   Real-time messaging requires a service that can quickly send and receive messages between components.

2. Step 2: Identify Google Cloud services for messaging

   Pub/Sub is designed for real-time messaging, while BigQuery is for analytics, Cloud Storage for file storage, and Cloud Functions for event-driven compute.

3. Final Answer:

   Pub/Sub -> Option C

4. Quick Check:

   Real-time messaging = Pub/Sub [OK]

Hint: Real-time messaging in GCP? Think Pub/Sub first [OK]

Common Mistakes:

• Confusing BigQuery as a messaging service
• Choosing Cloud Storage for streaming data
• Selecting Cloud Functions instead of messaging service

2. Which of the following is the correct syntax to create a Dataflow pipeline in Python using Apache Beam?

easy

A. pipeline = beam.Pipeline(options=options)

B. pipeline = Dataflow(options)

C. pipeline = beam.Create(options)

D. pipeline = DataflowPipeline(options=options)

Solution

1. Step 1: Recall Apache Beam pipeline creation syntax

   Apache Beam pipelines are created using beam.Pipeline with options passed as a parameter.

2. Step 2: Compare options

   pipeline = beam.Pipeline(options=options) matches the correct syntax. Other options use incorrect class names or methods.

3. Final Answer:

   pipeline = beam.Pipeline(options=options) -> Option A

4. Quick Check:

   Apache Beam pipeline creation = beam.Pipeline [OK]

Hint: Beam pipelines start with beam.Pipeline(options=...) [OK]

Common Mistakes:

• Using Dataflow instead of beam.Pipeline
• Confusing Create with pipeline creation
• Incorrect class names like DataflowPipeline

3. Given this simplified Dataflow pipeline code snippet, what will be the output in BigQuery?

p = beam.Pipeline(options=options)
(p
 | 'Read' >> beam.io.ReadFromPubSub(subscription='sub')
 | 'Parse' >> beam.Map(lambda x: x.decode('utf-8'))
 | 'Write' >> beam.io.WriteToBigQuery('project:dataset.table'))
p.run()

medium

A. Data will be stored in Cloud Storage instead of BigQuery

B. Data from Pub/Sub messages decoded and stored in BigQuery table

C. Pipeline will fail due to missing schema in WriteToBigQuery

D. Raw Pub/Sub messages stored as bytes in BigQuery

Solution

1. Step 1: Analyze pipeline steps

   The pipeline reads messages from Pub/Sub, decodes bytes to strings, then writes to BigQuery.

2. Step 2: Understand output behavior

   Since messages are decoded before writing, the data stored in BigQuery will be the decoded string form.

3. Final Answer:

   Data from Pub/Sub messages decoded and stored in BigQuery table -> Option B

4. Quick Check:

   Decoded Pub/Sub data written to BigQuery = Data from Pub/Sub messages decoded and stored in BigQuery table [OK]

Hint: Decoded Pub/Sub data goes to BigQuery if WriteToBigQuery used [OK]

Common Mistakes:

• Assuming raw bytes are stored without decoding
• Thinking pipeline fails without explicit schema (default schema can be inferred)
• Confusing output destination as Cloud Storage

4. You have a Dataflow pipeline that reads from Pub/Sub but fails with a permission error. What is the most likely cause?

medium

A. The Pub/Sub topic does not exist

B. BigQuery dataset is not created

C. The Dataflow pipeline code has syntax errors

D. The service account running Dataflow lacks Pub/Sub Subscriber role

Solution

1. Step 1: Identify permission error context

   Permission errors usually mean the service account lacks rights to access a resource.

2. Step 2: Match error to resource

   Since the pipeline reads from Pub/Sub, the service account needs the Subscriber role on the Pub/Sub topic.

3. Final Answer:

   The service account running Dataflow lacks Pub/Sub Subscriber role -> Option D

4. Quick Check:

   Permission error reading Pub/Sub = missing Subscriber role [OK]

Hint: Permission errors often mean missing roles on resources [OK]

Common Mistakes:

• Assuming syntax errors cause permission errors
• Thinking topic absence causes permission errors
• Confusing BigQuery dataset issues with Pub/Sub permissions

5. You want to build a data pipeline that ingests streaming IoT sensor data, processes it in real-time, and stores aggregated results hourly in BigQuery. Which pattern best fits this use case?

hard

A. Use Pub/Sub for ingestion, Dataflow streaming for processing with windowing, then write to BigQuery

B. Use Cloud Storage for ingestion, batch Dataflow jobs for processing, then write to BigQuery

C. Use BigQuery streaming inserts directly from IoT devices

D. Use Cloud Functions triggered by Cloud Storage uploads to process data

Solution

1. Step 1: Identify streaming ingestion and processing needs

   IoT sensor data is streaming and requires real-time processing with time-based aggregation.

2. Step 2: Match GCP services to requirements

   Pub/Sub handles streaming ingestion, Dataflow streaming supports windowed aggregation, and BigQuery stores results efficiently.

3. Final Answer:

   Use Pub/Sub for ingestion, Dataflow streaming for processing with windowing, then write to BigQuery -> Option A

4. Quick Check:

   Streaming ingestion + windowed processing + BigQuery = Use Pub/Sub for ingestion, Dataflow streaming for processing with windowing, then write to BigQuery [OK]

Hint: Streaming + windowed aggregation = Pub/Sub + Dataflow + BigQuery [OK]

Common Mistakes:

• Using batch processing for streaming data
• Trying to write streaming data directly to BigQuery without processing
• Using Cloud Storage which is for batch files, not streaming