Task groups for visual organization in Apache Airflow - Mini Project: Build & Apply

Task groups for visual organization

📖 Scenario: You are managing a data pipeline in Apache Airflow. Your pipeline has multiple related tasks that you want to group visually to keep the workflow clear and organized.

🎯 Goal: Build an Airflow DAG that uses TaskGroup to visually group related tasks together.

📋 What You'll Learn

Create a DAG named example_task_group_dag with default arguments

Create a TaskGroup named processing_tasks

Inside the processing_tasks group, create three DummyOperator tasks named extract, transform, and load

Set the task dependencies so that extract runs before transform, and transform runs before load

Print the DAG structure at the end

💡 Why This Matters

🌍 Real World

TaskGroups help organize complex workflows visually in Airflow UI, making pipelines easier to understand and maintain.

💼 Career

Understanding TaskGroups is important for data engineers and DevOps professionals managing scalable and readable data pipelines.

1

Create the DAG and import required modules

Import DAG, DummyOperator, and TaskGroup from airflow and airflow.operators.dummy. Create a DAG named example_task_group_dag with start_date set to datetime(2024, 1, 1) and schedule_interval set to None.

Apache Airflow

# Import DAG, DummyOperator, TaskGroup
# Create DAG named example_task_group_dag with start_date=datetime(2024, 1, 1) and schedule_interval=None
# Your code here

Need a hint?

Use from airflow import DAG and from airflow.operators.dummy import DummyOperator. Use datetime from the datetime module for the start date.

2

Create a TaskGroup named processing_tasks

Inside the DAG context, create a TaskGroup named processing_tasks.

Apache Airflow

from airflow import DAG
from airflow.operators.dummy import DummyOperator
from airflow.utils.task_group import TaskGroup
from datetime import datetime

dag = DAG(
    'example_task_group_dag',
    start_date=datetime(2024, 1, 1),
    schedule_interval=None
)

# Create a TaskGroup named processing_tasks
# Your code here

Need a hint?

Create the task group by calling TaskGroup('processing_tasks', dag=dag).

3

Create tasks inside the TaskGroup and set dependencies

Inside the processing_tasks group, create three DummyOperator tasks named extract, transform, and load. Set the dependencies so that extract runs before transform, and transform runs before load.

Apache Airflow

from airflow import DAG
from airflow.operators.dummy import DummyOperator
from airflow.utils.task_group import TaskGroup
from datetime import datetime

dag = DAG(
    'example_task_group_dag',
    start_date=datetime(2024, 1, 1),
    schedule_interval=None
)

processing_tasks = TaskGroup('processing_tasks', dag=dag)

# Create DummyOperator tasks extract, transform, load inside processing_tasks
# Set extract >> transform >> load dependencies
# Your code here

Need a hint?

Create each task with DummyOperator and assign task_group=processing_tasks. Use >> to set dependencies.

4

Print the DAG structure

Print the DAG structure by calling print(dag.task_dict).

Apache Airflow

from airflow import DAG
from airflow.operators.dummy import DummyOperator
from airflow.utils.task_group import TaskGroup
from datetime import datetime

dag = DAG(
    'example_task_group_dag',
    start_date=datetime(2024, 1, 1),
    schedule_interval=None
)

processing_tasks = TaskGroup('processing_tasks', dag=dag)

extract = DummyOperator(task_id='extract', task_group=processing_tasks)
transform = DummyOperator(task_id='transform', task_group=processing_tasks)
load = DummyOperator(task_id='load', task_group=processing_tasks)

extract >> transform >> load

# Print the DAG task dictionary
# Your code here

Need a hint?

Use print(dag.task_dict) to see the tasks including those inside the task group.