Azure operators in Apache Airflow - Practice Problems & Coding Challenges

Challenge - 5 Problems

💻 Command Output

intermediate

Output of Azure Data Factory Pipeline Run Operator

What is the output of the following Airflow task using AzureDataFactoryRunPipelineOperator when the pipeline run is successful?

Apache Airflow

run_pipeline = AzureDataFactoryRunPipelineOperator(
    task_id='run_pipeline',
    pipeline_name='example_pipeline',
    azure_data_factory_conn_id='azure_data_factory_default',
    resource_group_name='my_resource_group',
    factory_name='my_factory'
)

result = run_pipeline.execute({})
print(result)

A Returns a dictionary containing the pipeline run ID and status 'Succeeded'

B Returns None because execute method does not return anything

C Raises AirflowException due to missing pipeline parameters

D Returns a string with the pipeline name only

💡 Hint

Think about what the operator returns after triggering a pipeline run.

The AzureDataFactoryRunPipelineOperator triggers a pipeline and returns a dictionary with details including the run ID and status. This helps track the pipeline execution.

❓ Troubleshoot

intermediate

Troubleshooting Azure Blob Storage Hook Connection Error

You have an Airflow task using AzureBlobStorageHook but it fails with a connection error. Which of the following is the most likely cause?

Apache Airflow

hook = AzureBlobStorageHook(azure_blob_conn_id='azure_blob_default')
container_client = hook.get_container_client('mycontainer')
container_client.list_blobs()

A The hook does not support listing blobs in a container

B The container name 'mycontainer' does not exist in Azure Blob Storage

C The Azure connection ID 'azure_blob_default' is not configured or has wrong credentials

D The Airflow worker does not have internet access

💡 Hint

Check the connection configuration in Airflow UI.

A connection error usually means the credentials or connection ID is missing or incorrect in Airflow's connection settings.

❓ Configuration

advanced

Configuring Azure Data Lake Storage Gen2 Hook with OAuth

Which configuration snippet correctly sets up AzureDataLakeStorageHook to authenticate using OAuth with a service principal?

A

hook = AzureDataLakeStorageHook(
    account_name='storage-account',
    sas_token='sas-token'
)

B

hook = AzureDataLakeStorageHook(
    account_name='storage-account',
    auth_type='SharedKey',
    client_id='client-id'
)

C

hook = AzureDataLakeStorageHook(
    connection_string='DefaultEndpointsProtocol=https;AccountName=storage-account;AccountKey=key;EndpointSuffix=core.windows.net'
)

D

hook = AzureDataLakeStorageHook(
    tenant_id='tenant-id',
    client_id='client-id',
    client_secret='client-secret',
    account_name='storage-account',
    auth_type='OAuth'
)

💡 Hint

OAuth requires tenant ID, client ID, and client secret.

OAuth authentication for Azure Data Lake Storage Gen2 requires tenant ID, client ID, client secret, and specifying auth_type as 'OAuth'.

🔀 Workflow

advanced

Order of Tasks for Uploading File to Azure Blob Storage and Triggering Data Factory Pipeline

Given these Airflow tasks, which option shows the correct order to upload a file to Azure Blob Storage and then trigger an Azure Data Factory pipeline that processes the file?

A 3,1,2,4

B 3,2,1,4

C 1,3,2,4

D 2,1,3,4

💡 Hint

The file must be uploaded before the pipeline runs.

The workflow should start, upload the file, then trigger the pipeline, and finally end.

✅ Best Practice

expert

Best Practice for Handling Secrets in Azure Operators within Airflow

Which option describes the best practice for managing Azure credentials securely when using Azure operators in Airflow?

A Hardcode client secrets directly in the DAG Python files for easy access

B Store credentials in Airflow Connections and use environment variables or Azure Key Vault integration to avoid hardcoding secrets

C Use plain text files on the Airflow server to store Azure credentials and read them in the DAG

D Embed credentials in the operator parameters directly in the DAG code

💡 Hint

Think about security and secret management best practices.

Storing secrets in Airflow Connections and integrating with Azure Key Vault or environment variables keeps credentials secure and avoids exposure in code.

Practice

1. What is the main purpose of Azure operators in Apache Airflow?

easy

A. To automate and manage Azure cloud services within Airflow workflows

B. To create user interfaces for Azure services

C. To replace Azure CLI commands manually

D. To monitor local server hardware performance

Solution

1. Step 1: Understand Azure operators role

   Azure operators are designed to automate tasks on Azure cloud services from Airflow.

2. Step 2: Compare options with this role

   Only To automate and manage Azure cloud services within Airflow workflows describes automation and management of Azure services, which matches the purpose.

3. Final Answer:

   To automate and manage Azure cloud services within Airflow workflows -> Option A

4. Quick Check:

   Azure operators automate Azure tasks = C [OK]

Hint: Azure operators automate Azure tasks in Airflow [OK]

Common Mistakes:

• Confusing operators with UI tools
• Thinking operators replace manual CLI commands only
• Assuming operators monitor local hardware

2. Which of the following is the correct way to import the Azure Data Factory operator in Airflow?

easy

A. import AzureDataFactoryRunPipelineOperator from airflow.microsoft.azure

B. from airflow.providers.microsoft.azure.operators.data_factory import AzureDataFactoryRunPipelineOperator

C. from airflow.operators.azure.datafactory import RunPipelineOperator

D. from airflow.providers.azure.operators.datafactory import AzureRunPipelineOperator

Solution

1. Step 1: Recall correct import path for Azure Data Factory operator

   The official import path uses 'airflow.providers.microsoft.azure.operators.data_factory' and the class name 'AzureDataFactoryRunPipelineOperator'.

2. Step 2: Check each option for syntax and correctness

   from airflow.providers.microsoft.azure.operators.data_factory import AzureDataFactoryRunPipelineOperator matches the correct module and class name exactly; others have wrong paths or class names.

3. Final Answer:

   from airflow.providers.microsoft.azure.operators.data_factory import AzureDataFactoryRunPipelineOperator -> Option B

4. Quick Check:

   Correct import path and class name = A [OK]

Hint: Use full provider path with correct class name [OK]

Common Mistakes:

• Using incomplete or wrong import paths
• Incorrect class names or module names
• Mixing provider namespaces

3. Given this Airflow task using Azure VM operator:

run_vm = AzureVirtualMachineStartOperator(
    task_id='start_vm',
    resource_group_name='mygroup',
    vm_name='myvm'
)

What will happen when this task runs?

medium

A. It will create a new virtual machine named 'myvm'

B. It will stop the Azure virtual machine named 'myvm' in 'mygroup'

C. It will start the Azure virtual machine named 'myvm' in 'mygroup'

D. It will delete the virtual machine named 'myvm'

Solution

1. Step 1: Identify operator purpose

   The AzureVirtualMachineStartOperator is designed to start an existing VM.

2. Step 2: Analyze parameters and action

   The task specifies resource group and VM name, so it will start that VM, not stop, create, or delete it.

3. Final Answer:

   It will start the Azure virtual machine named 'myvm' in 'mygroup' -> Option C

4. Quick Check:

   Start operator starts VM = A [OK]

Hint: Operator name shows action: Start means start VM [OK]

Common Mistakes:

• Confusing start with stop or delete operators
• Assuming it creates a VM
• Ignoring resource group parameter

4. You wrote this Airflow task to delete an Azure storage container:

delete_container = AzureStorageDeleteOperator(
    task_id='delete_container',
    container_name='data',
    connection_string='my_conn_str'
)

But the task fails with an error about missing parameters. What is the likely cause?

medium

A. The operator does not support deleting containers, only blobs

B. The operator requires 'account_name' and 'account_key' instead of 'connection_string'

C. The operator requires 'resource_group_name' parameter

D. The operator requires the 'blob_name' parameter to delete a container

Solution

1. Step 1: Understand AzureStorageDeleteOperator capabilities

   This operator deletes blobs inside containers, not containers themselves.

2. Step 2: Analyze error cause

   Trying to delete a container causes failure because container deletion is unsupported by this operator.

3. Final Answer:

   The operator does not support deleting containers, only blobs -> Option A

4. Quick Check:

   Operator deletes blobs, not containers = D [OK]

Hint: Check operator docs for supported resource types [OK]

Common Mistakes:

• Assuming container deletion is supported
• Confusing connection parameters
• Adding unrelated parameters

5. You want to run an Azure Data Factory pipeline from Airflow and wait for its completion. Which operator and parameters combination is correct?

hard

A. AzureDataFactoryRunPipelineOperator with parameters: pipeline_id, resource_group, factory, async=True

B. AzureDataFactoryRunOperator with parameters: pipeline_name, resource_group_name, factory_name, wait_for_completion=True

C. AzureDataFactoryPipelineOperator with parameters: pipeline_name, resource_group_name, factory_name, wait=True

D. AzureDataFactoryRunPipelineOperator with parameters: pipeline_name, resource_group_name, factory_name, wait_for_termination=True

Solution

1. Step 1: Identify correct operator and parameters

   The official operator is AzureDataFactoryRunPipelineOperator. It uses 'pipeline_name', 'resource_group_name', 'factory_name', and 'wait_for_termination' to wait for completion.

2. Step 2: Compare options for exact names and flags

   AzureDataFactoryRunPipelineOperator with parameters: pipeline_name, resource_group_name, factory_name, wait_for_termination=True matches the correct operator and parameters exactly; others have wrong names or parameters.

3. Final Answer:

   AzureDataFactoryRunPipelineOperator with parameters: pipeline_name, resource_group_name, factory_name, wait_for_termination=True -> Option D

4. Quick Check:

   Correct operator and wait param = B [OK]

Hint: Use 'wait_for_termination=True' to wait for pipeline finish [OK]

Common Mistakes:

• Using wrong operator class names
• Incorrect parameter names
• Confusing async flags