Task groups for visual organization in Apache Airflow - Commands & Configuration

Introduction

When you have many tasks in an Airflow workflow, it can get messy and hard to understand. Task groups help by grouping related tasks together visually, making the workflow easier to read and manage.

When your workflow has many tasks that belong to the same step or process.

When you want to make your Airflow UI cleaner and easier to navigate.

When you want to reuse or organize tasks logically without changing their execution.

When you want to show a clear structure of your workflow to your team.

When you want to simplify complex dependencies by grouping tasks.

Config File - my_dag.py

my_dag.py

from airflow import DAG
from airflow.operators.bash import BashOperator
from airflow.utils.task_group import TaskGroup
from datetime import datetime

default_args = {
    'start_date': datetime(2024, 1, 1),
}

with DAG('example_task_group_dag', default_args=default_args, schedule_interval='@daily', catchup=False) as dag:
    start = BashOperator(
        task_id='start',
        bash_command='echo Start'
    )

    with TaskGroup('group1', tooltip='Tasks for step 1') as group1:
        task1 = BashOperator(
            task_id='task1',
            bash_command='echo Task 1'
        )
        task2 = BashOperator(
            task_id='task2',
            bash_command='echo Task 2'
        )

    with TaskGroup('group2', tooltip='Tasks for step 2') as group2:
        task3 = BashOperator(
            task_id='task3',
            bash_command='echo Task 3'
        )
        task4 = BashOperator(
            task_id='task4',
            bash_command='echo Task 4'
        )

    end = BashOperator(
        task_id='end',
        bash_command='echo End'
    )

    start >> group1 >> group2 >> end

This DAG defines a simple workflow with a start and end task.

Two TaskGroup blocks group related tasks: group1 contains task1 and task2, and group2 contains task3 and task4.

The groups help visually organize tasks in the Airflow UI, showing them as collapsible sections with tooltips.

The dependencies connect the start task to group1, then group2, then the end task, preserving execution order.

Commands

List all available DAGs to verify that the new DAG is recognized by Airflow.

Terminal

airflow dags list

Expected Output

example_task_group_dag

Trigger the DAG run manually to start the workflow and see the task groups in action.

Terminal

airflow dags trigger example_task_group_dag

Expected Output

Created <DagRun example_task_group_dag @ 2024-06-01 12:00:00: manual__2024-06-01T12:00:00+00:00, externally triggered: True>

List all tasks in the DAG, including those inside task groups, to confirm their presence.

Terminal

airflow tasks list example_task_group_dag

Expected Output

start

group1.task1

group1.task2

group2.task3

group2.task4

end

Test run a single task inside a task group to verify it runs correctly without executing the whole DAG.

Terminal

airflow tasks test example_task_group_dag group1.task1 2024-06-01

Expected Output

[2024-06-01 00:00:00,000] {bash.py:123} INFO - Running command: echo Task 1

[2024-06-01 00:00:00,001] {bash.py:130} INFO - Output: Task 1

[2024-06-01 00:00:00,002] {bash.py:135} INFO - Command exited with return code 0

Key Concept

If you remember nothing else from this pattern, remember: Task groups help you organize and visually simplify complex Airflow workflows without changing task execution.

Common Mistakes

Using task groups but not prefixing task IDs inside groups.

Task IDs inside groups must be unique and are prefixed automatically; forgetting this can cause confusion or conflicts.

Always refer to tasks inside groups with the full ID including the group prefix, like 'group1.task1'.

Trying to set dependencies directly between tasks in different groups without using full task IDs.

Dependencies must use full task IDs including group prefixes to work correctly.

Use full task IDs like 'group1.task2 >> group2.task3' to set dependencies between tasks in different groups.

Summary

Define TaskGroup blocks in your DAG to group related tasks visually.

Use full task IDs with group prefixes to manage dependencies and task references.

Trigger and test your DAG to see task groups as collapsible sections in the Airflow UI.