Why cloud operators simplify infrastructure tasks in Apache Airflow - See It in Action

Why Cloud Operators Simplify Infrastructure Tasks

📖 Scenario: You are working as a DevOps engineer managing workflows in Apache Airflow. You want to automate infrastructure tasks like creating and deleting cloud resources using Airflow's cloud operators. This project will help you understand how cloud operators simplify these tasks by wrapping complex API calls into easy-to-use Python classes.

🎯 Goal: Build a simple Airflow DAG that uses a cloud operator to create a cloud storage bucket and then delete it. This will show how cloud operators reduce manual scripting and make infrastructure automation easier.

📋 What You'll Learn

Create a Python dictionary called default_args with Airflow DAG default parameters

Create a DAG object called cloud_operator_dag with the ID 'cloud_operator_demo'

Use the GCSCreateBucketOperator to create a bucket named 'my-test-bucket'

Use the GCSDeleteBucketOperator to delete the bucket named 'my-test-bucket'

Set task dependencies so the bucket is created before it is deleted

Print the DAG ID at the end

💡 Why This Matters

🌍 Real World

DevOps engineers use cloud operators in Airflow to automate cloud infrastructure tasks without writing complex API code.

💼 Career

Knowing how to use cloud operators helps you automate and manage cloud resources efficiently, a key skill for DevOps and cloud engineering roles.

1

Set up Airflow DAG default arguments

Create a dictionary called default_args with these exact entries: 'owner': 'airflow', 'start_date': datetime(2024, 1, 1), and 'retries': 1. Import datetime from the datetime module.

Apache Airflow

from datetime import datetime

# Create default_args dictionary with owner, start_date, and retries
# Your code here

Need a hint?

Use default_args = {'owner': 'airflow', 'start_date': datetime(2024, 1, 1), 'retries': 1}.

2

Create the Airflow DAG object

Import DAG from airflow. Create a DAG object called cloud_operator_dag with the ID 'cloud_operator_demo', using default_args and a schedule interval of '@daily'.

Apache Airflow

from airflow import DAG

# Create DAG object cloud_operator_dag with dag_id 'cloud_operator_demo', default_args, and schedule_interval '@daily'
# Your code here

Need a hint?

Use cloud_operator_dag = DAG(dag_id='cloud_operator_demo', default_args=default_args, schedule_interval='@daily').

3

Add cloud operators to create and delete a bucket

Import GCSCreateBucketOperator and GCSDeleteBucketOperator from airflow.providers.google.cloud.operators.gcs. Create two tasks inside cloud_operator_dag: create_bucket using GCSCreateBucketOperator to create a bucket named 'my-test-bucket', and delete_bucket using GCSDeleteBucketOperator to delete the same bucket. Set create_bucket to run before delete_bucket.

Apache Airflow

from airflow.providers.google.cloud.operators.gcs import GCSCreateBucketOperator, GCSDeleteBucketOperator

# Create create_bucket task with GCSCreateBucketOperator to create 'my-test-bucket'
# Create delete_bucket task with GCSDeleteBucketOperator to delete 'my-test-bucket'
# Set create_bucket to run before delete_bucket
# Your code here

Need a hint?

Use create_bucket = GCSCreateBucketOperator(task_id='create_bucket', bucket_name='my-test-bucket', dag=cloud_operator_dag) and similarly for delete_bucket. Then set create_bucket >> delete_bucket.

4

Print the DAG ID

Write a print statement to display the DAG ID of cloud_operator_dag.

Apache Airflow

print(cloud_operator_dag.dag_id)  # Print the DAG ID
# Your code here

Need a hint?

Use print(cloud_operator_dag.dag_id) to show the DAG ID.