Azure operators in Apache Airflow - Mini Project: Build & Apply

Using Azure Operators in Airflow

📖 Scenario: You are working as a data engineer. Your team uses Apache Airflow to automate cloud tasks. You want to create a simple workflow that uses Azure operators to manage Azure resources.

🎯 Goal: Build an Airflow DAG that uses Azure operators to create a resource group and then delete it. This will help you understand how to use Azure operators in Airflow.

📋 What You'll Learn

Create an Airflow DAG named azure_resource_group_dag

Use AzureResourceGroupCreateOperator to create a resource group

Use AzureResourceGroupDeleteOperator to delete the resource group

Set the resource group name to my_test_resource_group

Set the location to eastus

Set task dependencies so creation runs before deletion

💡 Why This Matters

🌍 Real World

Automating Azure resource management with Airflow helps teams save time and avoid manual errors.

💼 Career

Many cloud and data engineering roles require knowledge of Airflow and Azure automation.

1

Create the Airflow DAG and import Azure operators

Create a DAG named azure_resource_group_dag with default arguments. Import AzureResourceGroupCreateOperator and AzureResourceGroupDeleteOperator from airflow.providers.microsoft.azure.operators.resource_group. Set the DAG to run once and start on 2024-01-01.

Apache Airflow

# Import DAG and Azure operators
# Create DAG named azure_resource_group_dag with start_date 2024-01-01 and schedule_interval '@once'
# Your code here

Hint

Use from airflow import DAG and import the Azure operators from airflow.providers.microsoft.azure.operators.resource_group.

2

Add resource group name and location variables

Create two variables: resource_group_name set to 'my_test_resource_group' and location set to 'eastus'.

Apache Airflow

from airflow import DAG
from airflow.providers.microsoft.azure.operators.resource_group import AzureResourceGroupCreateOperator, AzureResourceGroupDeleteOperator
from datetime import datetime

default_args = {
    'start_date': datetime(2024, 1, 1)
}

dag = DAG(
    'azure_resource_group_dag',
    default_args=default_args,
    schedule_interval='@once'
)

# Create variables resource_group_name and location
# Your code here

Hint

Define two string variables exactly as named and valued.

3

Add tasks to create and delete the resource group

Add two tasks to the DAG: create_rg using AzureResourceGroupCreateOperator with resource_group_name and location, and delete_rg using AzureResourceGroupDeleteOperator with resource_group_name. Assign the DAG to both tasks.

Apache Airflow

from airflow import DAG
from airflow.providers.microsoft.azure.operators.resource_group import AzureResourceGroupCreateOperator, AzureResourceGroupDeleteOperator
from datetime import datetime

default_args = {
    'start_date': datetime(2024, 1, 1)
}

dag = DAG(
    'azure_resource_group_dag',
    default_args=default_args,
    schedule_interval='@once'
)

resource_group_name = 'my_test_resource_group'
location = 'eastus'

# Create create_rg and delete_rg tasks using AzureResourceGroupCreateOperator and AzureResourceGroupDeleteOperator
# Your code here

Hint

Use the operator classes with correct parameters and assign the DAG to each task.

4

Set task dependencies and print confirmation

Set the task dependency so create_rg runs before delete_rg. Then add a print statement that outputs exactly 'DAG setup complete: Resource group will be created and then deleted.'

Apache Airflow

from airflow import DAG
from airflow.providers.microsoft.azure.operators.resource_group import AzureResourceGroupCreateOperator, AzureResourceGroupDeleteOperator
from datetime import datetime

default_args = {
    'start_date': datetime(2024, 1, 1)
}

dag = DAG(
    'azure_resource_group_dag',
    default_args=default_args,
    schedule_interval='@once'
)

resource_group_name = 'my_test_resource_group'
location = 'eastus'

create_rg = AzureResourceGroupCreateOperator(
    task_id='create_rg',
    resource_group_name=resource_group_name,
    location=location,
    dag=dag
)

delete_rg = AzureResourceGroupDeleteOperator(
    task_id='delete_rg',
    resource_group_name=resource_group_name,
    dag=dag
)

# Set create_rg to run before delete_rg
# Print confirmation message
# Your code here

Hint

Use the bitshift operator >> to set task order. Use print() for the message.

Practice

1. What is the main purpose of Azure operators in Apache Airflow?

easy

A. To automate and manage Azure cloud services within Airflow workflows

B. To create user interfaces for Azure services

C. To replace Azure CLI commands manually

D. To monitor local server hardware performance

Solution

1. Step 1: Understand Azure operators role

   Azure operators are designed to automate tasks on Azure cloud services from Airflow.

2. Step 2: Compare options with this role

   Only To automate and manage Azure cloud services within Airflow workflows describes automation and management of Azure services, which matches the purpose.

3. Final Answer:

   To automate and manage Azure cloud services within Airflow workflows -> Option A

4. Quick Check:

   Azure operators automate Azure tasks = C [OK]

Hint: Azure operators automate Azure tasks in Airflow [OK]

Common Mistakes:

• Confusing operators with UI tools
• Thinking operators replace manual CLI commands only
• Assuming operators monitor local hardware

2. Which of the following is the correct way to import the Azure Data Factory operator in Airflow?

easy

A. import AzureDataFactoryRunPipelineOperator from airflow.microsoft.azure

B. from airflow.providers.microsoft.azure.operators.data_factory import AzureDataFactoryRunPipelineOperator

C. from airflow.operators.azure.datafactory import RunPipelineOperator

D. from airflow.providers.azure.operators.datafactory import AzureRunPipelineOperator

Solution

1. Step 1: Recall correct import path for Azure Data Factory operator

   The official import path uses 'airflow.providers.microsoft.azure.operators.data_factory' and the class name 'AzureDataFactoryRunPipelineOperator'.

2. Step 2: Check each option for syntax and correctness

   from airflow.providers.microsoft.azure.operators.data_factory import AzureDataFactoryRunPipelineOperator matches the correct module and class name exactly; others have wrong paths or class names.

3. Final Answer:

   from airflow.providers.microsoft.azure.operators.data_factory import AzureDataFactoryRunPipelineOperator -> Option B

4. Quick Check:

   Correct import path and class name = A [OK]

Hint: Use full provider path with correct class name [OK]

Common Mistakes:

• Using incomplete or wrong import paths
• Incorrect class names or module names
• Mixing provider namespaces

3. Given this Airflow task using Azure VM operator:

run_vm = AzureVirtualMachineStartOperator(
    task_id='start_vm',
    resource_group_name='mygroup',
    vm_name='myvm'
)

What will happen when this task runs?

medium

A. It will create a new virtual machine named 'myvm'

B. It will stop the Azure virtual machine named 'myvm' in 'mygroup'

C. It will start the Azure virtual machine named 'myvm' in 'mygroup'

D. It will delete the virtual machine named 'myvm'

Solution

1. Step 1: Identify operator purpose

   The AzureVirtualMachineStartOperator is designed to start an existing VM.

2. Step 2: Analyze parameters and action

   The task specifies resource group and VM name, so it will start that VM, not stop, create, or delete it.

3. Final Answer:

   It will start the Azure virtual machine named 'myvm' in 'mygroup' -> Option C

4. Quick Check:

   Start operator starts VM = A [OK]

Hint: Operator name shows action: Start means start VM [OK]

Common Mistakes:

• Confusing start with stop or delete operators
• Assuming it creates a VM
• Ignoring resource group parameter

4. You wrote this Airflow task to delete an Azure storage container:

delete_container = AzureStorageDeleteOperator(
    task_id='delete_container',
    container_name='data',
    connection_string='my_conn_str'
)

But the task fails with an error about missing parameters. What is the likely cause?

medium

A. The operator does not support deleting containers, only blobs

B. The operator requires 'account_name' and 'account_key' instead of 'connection_string'

C. The operator requires 'resource_group_name' parameter

D. The operator requires the 'blob_name' parameter to delete a container

Solution

1. Step 1: Understand AzureStorageDeleteOperator capabilities

   This operator deletes blobs inside containers, not containers themselves.

2. Step 2: Analyze error cause

   Trying to delete a container causes failure because container deletion is unsupported by this operator.

3. Final Answer:

   The operator does not support deleting containers, only blobs -> Option A

4. Quick Check:

   Operator deletes blobs, not containers = D [OK]

Hint: Check operator docs for supported resource types [OK]

Common Mistakes:

• Assuming container deletion is supported
• Confusing connection parameters
• Adding unrelated parameters

5. You want to run an Azure Data Factory pipeline from Airflow and wait for its completion. Which operator and parameters combination is correct?

hard

A. AzureDataFactoryRunPipelineOperator with parameters: pipeline_id, resource_group, factory, async=True

B. AzureDataFactoryRunOperator with parameters: pipeline_name, resource_group_name, factory_name, wait_for_completion=True

C. AzureDataFactoryPipelineOperator with parameters: pipeline_name, resource_group_name, factory_name, wait=True

D. AzureDataFactoryRunPipelineOperator with parameters: pipeline_name, resource_group_name, factory_name, wait_for_termination=True

Solution

1. Step 1: Identify correct operator and parameters

   The official operator is AzureDataFactoryRunPipelineOperator. It uses 'pipeline_name', 'resource_group_name', 'factory_name', and 'wait_for_termination' to wait for completion.

2. Step 2: Compare options for exact names and flags

   AzureDataFactoryRunPipelineOperator with parameters: pipeline_name, resource_group_name, factory_name, wait_for_termination=True matches the correct operator and parameters exactly; others have wrong names or parameters.

3. Final Answer:

   AzureDataFactoryRunPipelineOperator with parameters: pipeline_name, resource_group_name, factory_name, wait_for_termination=True -> Option D

4. Quick Check:

   Correct operator and wait param = B [OK]

Hint: Use 'wait_for_termination=True' to wait for pipeline finish [OK]

Common Mistakes:

• Using wrong operator class names
• Incorrect parameter names
• Confusing async flags