Azure operators in Apache Airflow - Mini Project: Build & Apply

Using Azure Operators in Airflow

📖 Scenario: You are working as a data engineer. Your team uses Apache Airflow to automate cloud tasks. You want to create a simple workflow that uses Azure operators to manage Azure resources.

🎯 Goal: Build an Airflow DAG that uses Azure operators to create a resource group and then delete it. This will help you understand how to use Azure operators in Airflow.

📋 What You'll Learn

Create an Airflow DAG named azure_resource_group_dag

Use AzureResourceGroupCreateOperator to create a resource group

Use AzureResourceGroupDeleteOperator to delete the resource group

Set the resource group name to my_test_resource_group

Set the location to eastus

Set task dependencies so creation runs before deletion

💡 Why This Matters

🌍 Real World

Automating Azure resource management with Airflow helps teams save time and avoid manual errors.

💼 Career

Many cloud and data engineering roles require knowledge of Airflow and Azure automation.

1

Create the Airflow DAG and import Azure operators

Create a DAG named azure_resource_group_dag with default arguments. Import AzureResourceGroupCreateOperator and AzureResourceGroupDeleteOperator from airflow.providers.microsoft.azure.operators.resource_group. Set the DAG to run once and start on 2024-01-01.

Apache Airflow

# Import DAG and Azure operators
# Create DAG named azure_resource_group_dag with start_date 2024-01-01 and schedule_interval '@once'
# Your code here

Need a hint?

Use from airflow import DAG and import the Azure operators from airflow.providers.microsoft.azure.operators.resource_group.

2

Add resource group name and location variables

Create two variables: resource_group_name set to 'my_test_resource_group' and location set to 'eastus'.

Apache Airflow

from airflow import DAG
from airflow.providers.microsoft.azure.operators.resource_group import AzureResourceGroupCreateOperator, AzureResourceGroupDeleteOperator
from datetime import datetime

default_args = {
    'start_date': datetime(2024, 1, 1)
}

dag = DAG(
    'azure_resource_group_dag',
    default_args=default_args,
    schedule_interval='@once'
)

# Create variables resource_group_name and location
# Your code here

Need a hint?

Define two string variables exactly as named and valued.

3

Add tasks to create and delete the resource group

Add two tasks to the DAG: create_rg using AzureResourceGroupCreateOperator with resource_group_name and location, and delete_rg using AzureResourceGroupDeleteOperator with resource_group_name. Assign the DAG to both tasks.

Apache Airflow

from airflow import DAG
from airflow.providers.microsoft.azure.operators.resource_group import AzureResourceGroupCreateOperator, AzureResourceGroupDeleteOperator
from datetime import datetime

default_args = {
    'start_date': datetime(2024, 1, 1)
}

dag = DAG(
    'azure_resource_group_dag',
    default_args=default_args,
    schedule_interval='@once'
)

resource_group_name = 'my_test_resource_group'
location = 'eastus'

# Create create_rg and delete_rg tasks using AzureResourceGroupCreateOperator and AzureResourceGroupDeleteOperator
# Your code here

Need a hint?

Use the operator classes with correct parameters and assign the DAG to each task.

4

Set task dependencies and print confirmation

Set the task dependency so create_rg runs before delete_rg. Then add a print statement that outputs exactly 'DAG setup complete: Resource group will be created and then deleted.'

Apache Airflow

from airflow import DAG
from airflow.providers.microsoft.azure.operators.resource_group import AzureResourceGroupCreateOperator, AzureResourceGroupDeleteOperator
from datetime import datetime

default_args = {
    'start_date': datetime(2024, 1, 1)
}

dag = DAG(
    'azure_resource_group_dag',
    default_args=default_args,
    schedule_interval='@once'
)

resource_group_name = 'my_test_resource_group'
location = 'eastus'

create_rg = AzureResourceGroupCreateOperator(
    task_id='create_rg',
    resource_group_name=resource_group_name,
    location=location,
    dag=dag
)

delete_rg = AzureResourceGroupDeleteOperator(
    task_id='delete_rg',
    resource_group_name=resource_group_name,
    dag=dag
)

# Set create_rg to run before delete_rg
# Print confirmation message
# Your code here

Need a hint?

Use the bitshift operator >> to set task order. Use print() for the message.