Dynamic task generation with loops in Apache Airflow - Step-by-Step Execution

Process Flow - Dynamic task generation with loops

Start DAG definition

↓

Define task list

↓

For each item in list

↓

Create task dynamically

↓

Set task dependencies

↓

DAG ready to run

This flow shows how a loop creates multiple tasks dynamically in an Airflow DAG, then sets their order.

Execution Sample

Apache Airflow

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime

dag = DAG('loop_dag', start_date=datetime(2024,1,1))
tasks = []
for i in range(3):
    task = BashOperator(task_id=f'task_{i}', bash_command=f'echo {i}', dag=dag)
    tasks.append(task)

This code creates 3 BashOperator tasks dynamically in a loop inside an Airflow DAG.

Process Table

Step	Loop Index i	Action	Task Created	Task List Length
1	0	Create task with task_id 'task_0'	task_0	1
2	1	Create task with task_id 'task_1'	task_1	2
3	2	Create task with task_id 'task_2'	task_2	3
4	-	Loop ends after i=2	-	3

💡 Loop ends after i=2 because range(3) produces 0,1,2

Status Tracker

Variable	Start	After 1	After 2	After 3	Final
i	-	0	1	2	-
tasks	[]	[task_0]	[task_0, task_1]	[task_0, task_1, task_2]	[task_0, task_1, task_2]

Key Moments - 2 Insights

Why does the loop stop after creating 3 tasks?

Because the range(3) generates numbers 0, 1, and 2 only, so the loop runs exactly 3 times as shown in execution_table rows 1-3.

How do we know each task has a unique task_id?

The task_id is created using f'task_{i}', so each loop iteration uses a different i value, producing unique IDs as seen in the 'Task Created' column.

Visual Quiz - 3 Questions

Look at the execution table, what is the task_id created at step 2?

A task_2

B task_0

C task_1

D task_3

💡 Hint

Check the 'Task Created' column at step 2 in the execution_table.

At which step does the loop end according to the execution table?

A Step 4

B Step 2

C Step 3

D Step 1

💡 Hint

Look for the row where the action says 'Loop ends' in the execution_table.

If range(5) was used instead of range(3), how many tasks would be in the tasks list after the loop?

A 4

B 5

C 3

D 6

💡 Hint

Refer to variable_tracker for how tasks list length changes with loop iterations.

Concept Snapshot

Dynamic task generation in Airflow:
- Use a loop to create tasks dynamically
- Each task needs a unique task_id
- Append tasks to a list for dependency management
- Loop runs as many times as range() specifies
- Tasks are added to DAG automatically

Full Transcript

This visual execution shows how to generate multiple tasks dynamically in an Airflow DAG using a loop. The loop runs three times, creating tasks with unique IDs 'task_0', 'task_1', and 'task_2'. Each task is appended to a list for later use. The loop stops after the third iteration because range(3) produces numbers 0, 1, and 2. This method helps create many similar tasks without writing repetitive code.