GCP operators (BigQuery, GCS, Dataflow) in Apache Airflow - Step-by-Step Execution

Process Flow - GCP operators (BigQuery, GCS, Dataflow)

Start DAG

↓

Trigger GCS Operator

↓

Trigger BigQuery Operator

↓

Trigger Dataflow Operator

↓

Check Task Status

↓

Complete DAG

The DAG starts, triggers GCS, then BigQuery, then Dataflow operators in order, checks their status, and completes.

Execution Sample

Apache Airflow

from airflow import DAG
from airflow.providers.google.cloud.operators.bigquery import BigQueryInsertJobOperator
from airflow.providers.google.cloud.operators.gcs import GCSCreateBucketOperator
from airflow.providers.google.cloud.operators.dataflow import DataflowStartFlexTemplateOperator

with DAG('gcp_operators_dag', schedule_interval=None, start_date=None, catchup=False) as dag:
    create_bucket = GCSCreateBucketOperator(task_id='create_bucket', bucket_name='my-bucket')
    run_bq = BigQueryInsertJobOperator(task_id='run_bq', configuration={"query": {"query": "SELECT 1", "useLegacySql": False}})
    start_dataflow = DataflowStartFlexTemplateOperator(task_id='start_dataflow', body={"launchParameter": {"jobName": "my-job", "containerSpecGcsPath": "gs://dataflow-templates/latest/flex/Word_Count"}})

    create_bucket >> run_bq >> start_dataflow

This Airflow DAG creates a GCS bucket, runs a BigQuery query, then starts a Dataflow job in sequence.

Process Table

Step	Task Triggered	Action	Status	Next Task
1	create_bucket	Create GCS bucket 'my-bucket'	Success	run_bq
2	run_bq	Run BigQuery SQL query 'SELECT 1'	Success	start_dataflow
3	start_dataflow	Start Dataflow job 'my-job' with flex template	Success	None
4	DAG	All tasks completed successfully	Success	End

💡 All tasks executed successfully in order, DAG run completes.

Status Tracker

Variable	Start	After create_bucket	After run_bq	After start_dataflow
GCS Bucket	None	my-bucket created	my-bucket exists	my-bucket exists
BigQuery Job	None	None	Query job completed	Query job completed
Dataflow Job	None	None	None	Dataflow job started

Key Moments - 3 Insights

Why does the Dataflow operator run only after BigQuery finishes?

Because the DAG defines dependencies with 'create_bucket >> run_bq >> start_dataflow', so each task waits for the previous to succeed (see execution_table steps 2 and 3).

What happens if the GCS bucket creation fails?

The DAG stops and does not run BigQuery or Dataflow tasks, since downstream tasks depend on the success of previous ones (see execution_table step 1).

How does Airflow know the status of each GCP operator?

Each operator reports success or failure after execution, which Airflow tracks to decide whether to proceed (see 'Status' column in execution_table).

Visual Quiz - 3 Questions

Look at the execution_table, what is the status of the BigQuery task at step 2?

A Failed

B Success

C Running

D Skipped

💡 Hint

Check the 'Status' column for step 2 in the execution_table.

At which step does the Dataflow job start according to the execution_table?

A Step 1

B Step 2

C Step 3

D Step 4

💡 Hint

Look for 'start_dataflow' task in the 'Task Triggered' column.

If the GCS bucket creation failed, what would happen to the BigQuery task?

A It would be skipped

B It would run anyway

C It would run after Dataflow

D It would retry immediately

💡 Hint

Refer to key_moments about task dependencies and failure impact.

Concept Snapshot

GCP Operators in Airflow:
- GCSCreateBucketOperator creates buckets
- BigQueryInsertJobOperator runs SQL queries
- DataflowStartFlexTemplateOperator starts Dataflow jobs
- Define task order with >> to set dependencies
- Airflow tracks success/failure to control flow

Full Transcript

This visual execution shows an Airflow DAG using GCP operators: first creating a GCS bucket, then running a BigQuery query, and finally starting a Dataflow job. Each task waits for the previous to succeed before running. The execution table tracks each step's action and status. Variables like bucket existence and job completion update after each task. Key moments clarify why tasks run in order and what happens on failure. The quiz tests understanding of task status and dependencies. This helps beginners see how Airflow manages GCP tasks step-by-step.