Dynamic task generation with loops in Apache Airflow - Practice Problems & Coding Challenges

Challenge - 5 Problems

💻 Command Output

intermediate

Output of dynamically generated Airflow tasks with loop

Given the following Airflow DAG snippet, what will be the output when the DAG is parsed and tasks are listed?

Apache Airflow

from airflow import DAG
from airflow.operators.dummy import DummyOperator
from datetime import datetime

with DAG('loop_dag', start_date=datetime(2024, 1, 1)) as dag:
    tasks = []
    for i in range(3):
        task = DummyOperator(task_id=f'task_{i}')
        tasks.append(task)

print([task.task_id for task in tasks])

A ['task_0', 'task_1', 'task_2']

B ['task_1', 'task_2', 'task_3']

C ['task_0', 'task_1']

D ['task_0', 'task_1', 'task_2', 'task_3']

💡 Hint

Remember that range(3) generates numbers 0, 1, 2.

The loop runs 3 times with i values 0, 1, and 2, creating tasks with those IDs.

🔀 Workflow

intermediate

Correct order of dynamic task dependencies in Airflow

Arrange the following code lines to correctly create a chain of tasks dynamically in Airflow using a loop.

A 4,2,1,3

B 4,1,3,2

C 4,1,2,3

D 1,4,2,3

💡 Hint

The loop must start first, then create task, set dependency, then update previous_task.

The loop starts, creates a task, sets dependency from previous_task to current task, then updates previous_task for next iteration.

❓ Troubleshoot

advanced

Error caused by dynamic task creation in Airflow

What error will this Airflow DAG code raise when parsing, and why? from airflow import DAG from airflow.operators.dummy import DummyOperator from datetime import datetime with DAG('error_dag', start_date=datetime(2024, 1, 1)) as dag: for i in range(3): task = DummyOperator(task_id='task')

A No error, DAG parses successfully

B SyntaxError: invalid syntax

C TypeError: DummyOperator() missing required positional argument

D AirflowDagValidationError: Duplicate task_id 'task'

💡 Hint

Task IDs must be unique in an Airflow DAG.

All tasks have the same task_id 'task', which is not allowed and causes a validation error.

🧠 Conceptual

advanced

Best practice for dynamic task creation in Airflow

Which statement best describes a best practice when generating tasks dynamically in Airflow using loops?

A Always assign unique task_ids to each task to avoid conflicts.

B Reuse the same task_id for all tasks to simplify the DAG.

C Create tasks outside the DAG context manager to improve performance.

D Avoid using loops; create each task manually for clarity.

💡 Hint

Think about how Airflow identifies tasks uniquely.

Unique task_ids are required for Airflow to distinguish tasks and avoid errors.

✅ Best Practice

expert

Handling dynamic task dependencies with variable task counts

You want to create a dynamic chain of tasks in Airflow where the number of tasks is not fixed and depends on external input. Which approach ensures the chain is correctly built regardless of the number of tasks?

A Create all tasks first, then outside the loop set dependencies manually between tasks by their index.

B Initialize a variable previous_task as None before the loop, then inside the loop, if previous_task is not None, set previous_task >> current_task, and update previous_task to current_task.

C Use a fixed number of tasks and ignore the external input to avoid complexity.

D Set all tasks to run in parallel without dependencies to simplify the DAG.

💡 Hint

Think about how to link tasks dynamically when you don't know how many there will be.

Initializing previous_task as None and updating it inside the loop allows chaining tasks dynamically regardless of count.