GCP operators (BigQuery, GCS, Dataflow) in Apache Airflow - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is the purpose of the BigQueryOperator in Airflow?

The BigQueryOperator runs SQL queries on Google BigQuery. It helps automate data analysis and loading tasks in workflows.

beginner

How does the GoogleCloudStorageCreateBucketOperator help in Airflow workflows?

It creates a new bucket in Google Cloud Storage (GCS). This is useful for organizing and storing data files automatically.

intermediate

What is the role of DataflowTemplateOperator in Airflow?

It launches a Dataflow job from a pre-built template. This operator helps run data processing pipelines without writing code each time.

intermediate

Which Airflow operator would you use to load data from GCS to BigQuery?

The GCSToBigQueryOperator loads data stored in Google Cloud Storage into a BigQuery table.

beginner

Why is it important to set the correct GCP connection in Airflow when using GCP operators?

The GCP connection provides authentication and project details. Without it, Airflow cannot access GCP services securely.

Which Airflow operator runs SQL queries on BigQuery?

A GCSToBigQueryOperator

B GoogleCloudStorageCreateBucketOperator

C DataflowTemplateOperator

D BigQueryOperator

BigQueryOperator is designed to execute SQL queries on BigQuery.

What does the GoogleCloudStorageCreateBucketOperator do?

A Creates a BigQuery dataset

B Runs a Dataflow job

C Creates a GCS bucket

D Loads data from GCS to BigQuery

It creates a new bucket in Google Cloud Storage.

Which operator launches a Dataflow job from a template?

A DataflowTemplateOperator

B BigQueryOperator

C GCSToBigQueryOperator

D GoogleCloudStorageCreateBucketOperator

DataflowTemplateOperator runs Dataflow jobs using pre-built templates.

To move data from GCS to BigQuery, which operator is best?

A GCSToBigQueryOperator

B GoogleCloudStorageCreateBucketOperator

C BigQueryOperator

D DataflowTemplateOperator

GCSToBigQueryOperator loads data from GCS into BigQuery tables.

Why must you configure a GCP connection in Airflow for GCP operators?

A To schedule tasks

B To authenticate and access GCP services

C To write SQL queries

D To create DAGs

The GCP connection handles authentication and project info for secure access.

Explain how Airflow uses GCP operators to automate data workflows involving BigQuery, GCS, and Dataflow.

Think about each operator's role and how they connect in a data pipeline.

Describe the steps to set up an Airflow DAG that loads data from GCS into BigQuery using GCP operators.

Focus on the sequence of operators and configuration needed.