Azure operators in Apache Airflow - Step-by-Step Execution

Process Flow - Azure operators

Start DAG

↓

Trigger Azure Operator

↓

Authenticate with Azure

↓

Execute Azure Task

↓

Check Task Status

↓

Success?

No→Handle Failure

Yes↓

Complete DAG

This flow shows how an Airflow DAG triggers an Azure operator, which authenticates, runs a task on Azure, checks the result, and completes or handles failure.

Execution Sample

Apache Airflow

from airflow.providers.microsoft.azure.operators.adls import AzureDataLakeStorageCreateDirectoryOperator

create_dir = AzureDataLakeStorageCreateDirectoryOperator(
    task_id='create_dir',
    directory_name='myfolder',
    azure_data_lake_conn_id='azure_default'
)

This code creates an Airflow task that uses an Azure operator to create a directory in Azure Data Lake Storage.

Process Table

Step	Action	Azure Operator Method	Input Parameters	Result/Output
1	Start DAG run	N/A	N/A	DAG started
2	Trigger Azure operator task	execute	directory_name='myfolder', azure_data_lake_conn_id='azure_default'	Operator initialized
3	Authenticate with Azure	get_conn	azure_data_lake_conn_id='azure_default'	Authentication successful
4	Create directory in ADLS	create_directory	directory_name='myfolder'	Directory 'myfolder' created
5	Check operation status	check_status	N/A	Success
6	Mark task complete	N/A	N/A	Task succeeded
7	DAG completes	N/A	N/A	DAG run finished successfully

💡 DAG run ends after successful Azure operator task execution and directory creation.

Status Tracker

Variable	Start	After Step 2	After Step 3	After Step 4	After Step 5	Final
directory_name	N/A	'myfolder'	'myfolder'	'myfolder'	'myfolder'	'myfolder'
conn_id	N/A	'azure_default'	'azure_default'	'azure_default'	'azure_default'	'azure_default'
task_status	N/A	initialized	authenticated	directory created	success	success

Key Moments - 3 Insights

Why does the operator need a connection ID like 'azure_default'?

The connection ID tells the operator how to authenticate with Azure. Without it, the operator cannot access Azure services. See execution_table step 3 where authentication happens using 'azure_default'.

What happens if the directory already exists?

The operator may raise an error or skip creation depending on its implementation. This would cause the task to fail at step 4 or 5 in the execution_table, triggering failure handling.

Is the DAG run complete immediately after triggering the operator?

No, the DAG waits for the operator to finish its task successfully (step 6) before completing (step 7). This ensures the Azure task finished properly.

Visual Quiz - 3 Questions

Look at the execution_table, what is the result after step 4?

A Authentication successful

B Directory 'myfolder' created

C Task succeeded

D DAG started

💡 Hint

Check the 'Result/Output' column for step 4 in the execution_table.

At which step does the operator authenticate with Azure?

A Step 3

B Step 5

C Step 2

D Step 6

💡 Hint

Look for 'Authentication successful' in the Result/Output column.

If the connection ID was incorrect, which step would fail?

A Step 6

B Step 4

C Step 3

D Step 7

💡 Hint

Authentication happens at step 3; failure there means no access to Azure.

Concept Snapshot

Azure operators in Airflow:
- Use Azure connection ID for authentication
- Operators perform Azure tasks (e.g., create directory)
- DAG waits for operator success before completing
- Errors in Azure tasks cause DAG failure
- Common operators: ADLS, VM, Blob Storage
- Define tasks with operator classes and parameters

Full Transcript

This visual execution shows how an Airflow DAG uses an Azure operator to create a directory in Azure Data Lake Storage. The DAG starts, triggers the operator, which authenticates using a connection ID, performs the directory creation, checks the result, and then completes the DAG run. Variables like directory_name and task_status change as the operator progresses. Key points include the need for correct Azure authentication and that the DAG waits for the operator to finish before completing. The quiz questions reinforce understanding of the steps and outcomes.