Task groups for visual organization in Apache Airflow - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Purpose of Task Groups in Airflow

What is the main purpose of using Task Groups in an Airflow DAG?

A To replace the need for defining task dependencies between tasks

B To execute all tasks in the group simultaneously regardless of dependencies

C To visually organize related tasks into a single group for better readability in the Airflow UI

D To automatically retry failed tasks within the group without manual configuration

💡 Hint

Think about how Airflow UI shows tasks and how grouping helps.

Task Groups help organize related tasks visually in the Airflow UI, making complex DAGs easier to understand. They do not change execution behavior or dependencies.

💻 Command Output

intermediate

Output of Task Group Structure in Airflow UI

Given the following Airflow DAG snippet using Task Groups, what will be the visual structure shown in the Airflow UI?

Apache Airflow

from airflow import DAG
from airflow.operators.dummy import DummyOperator
from airflow.utils.task_group import TaskGroup
from datetime import datetime

default_args = {'start_date': datetime(2024, 1, 1)}

dag = DAG('example_dag', default_args=default_args, schedule_interval='@daily')

start = DummyOperator(task_id='start', dag=dag)
end = DummyOperator(task_id='end', dag=dag)

with TaskGroup('group1', dag=dag) as group1:
    t1 = DummyOperator(task_id='task1')
    t2 = DummyOperator(task_id='task2')

start >> group1 >> end

A The UI shows 'start' connected directly to 'task1' and 'task2' with no grouping

B The UI shows five separate tasks: 'start', 'group1', 'task1', 'task2', and 'end' all ungrouped

C The UI shows only 'start' and 'end' tasks because Task Groups hide tasks by default

D The UI shows three nodes: 'start', a collapsed group named 'group1' containing 'task1' and 'task2', and 'end' connected in sequence

💡 Hint

Task Groups create a collapsible group in the UI containing the tasks inside.

Task Groups appear as a single collapsible node in the Airflow UI. Inside, the grouped tasks are shown. The sequence is start → group1 → end.

❓ Configuration

advanced

Correct Task Group Definition Syntax

Which of the following code snippets correctly defines a Task Group named 'processing' with two dummy tasks inside in Airflow 2.5+?

A

processing = TaskGroup('processing')
with processing:
    task_a = DummyOperator(task_id='task_a')
    task_b = DummyOperator(task_id='task_b')

B

processing = TaskGroup('processing')
processing.task_a = DummyOperator(task_id='task_a')
processing.task_b = DummyOperator(task_id='task_b')

C

with TaskGroup('processing') as processing:
    task_a = DummyOperator(task_id='task_a')
    task_b = DummyOperator(task_id='task_b')

D

with TaskGroup('processing'):
    task_a = DummyOperator(task_id='task_a')
    task_b = DummyOperator(task_id='task_b')

💡 Hint

TaskGroup can be used as a context manager assigned to a variable.

The correct way is to create a TaskGroup instance, then use it as a context manager to define tasks inside. Option A does this correctly.

❓ Troubleshoot

advanced

Error When Using Task Groups Without DAG Context

What error will occur if you define a Task Group and tasks inside it without associating them with a DAG or DAG context?

A AirflowException: TaskGroup must be used within a DAG context

B TypeError: 'NoneType' object is not iterable

C AttributeError: 'TaskGroup' object has no attribute 'dag'

D ValueError: Task IDs must be unique within a DAG

💡 Hint

Task Groups require a DAG context to know where they belong.

Task Groups must be defined inside a DAG context or passed a DAG explicitly. Otherwise, Airflow raises an AirflowException indicating the missing DAG context.

🔀 Workflow

expert

Task Group Dependency Behavior

Consider two Task Groups, groupA and groupB, each with multiple tasks. If you set the dependency groupA >> groupB, what is the effect on the tasks inside these groups?

A Only the first task in groupA must complete before the first task in groupB starts

B All tasks in groupA must complete before any task in groupB starts

C Tasks in groupA and groupB run in parallel ignoring the dependency

D Only tasks with matching task_ids in both groups depend on each other

💡 Hint

Task Group dependencies apply to all tasks inside the groups.

Setting groupA >> groupB creates dependencies from every task in groupA to every task in groupB, so all groupA tasks must finish before groupB tasks start.

Practice

1. What is the main purpose of using TaskGroup in Airflow DAGs?

easy

A. To visually group related tasks for better organization

B. To execute tasks in parallel automatically

C. To replace the need for defining individual tasks

D. To schedule DAG runs at specific times

Solution

1. Step 1: Understand TaskGroup purpose

   TaskGroup is designed to group related tasks visually in the Airflow UI.

2. Step 2: Differentiate from other features

   It does not change execution behavior or scheduling but improves readability.

3. Final Answer:

   To visually group related tasks for better organization -> Option A

4. Quick Check:

   TaskGroup = Visual grouping [OK]

Hint: TaskGroup groups tasks visually, not for execution control [OK]

Common Mistakes:

• Thinking TaskGroup runs tasks in parallel
• Confusing TaskGroup with scheduling
• Assuming TaskGroup replaces tasks

2. Which of the following is the correct syntax to create a task group named 'processing' in Airflow?

easy

A. with TaskGroup('processing') as tg:

B. TaskGroup('processing'):

C. group = TaskGroup('processing')

D. with task_group('processing') as tg:

Solution

1. Step 1: Recall TaskGroup syntax

   The correct way to create a task group is using a context manager: with TaskGroup('name') as variable:

2. Step 2: Check option correctness

   with TaskGroup('processing') as tg: uses correct capitalization and syntax. TaskGroup('processing'): misses 'with'. group = TaskGroup('processing') misses 'with' and context manager. with task_group('processing') as tg: uses wrong function name casing.

3. Final Answer:

   with TaskGroup('processing') as tg: -> Option A

4. Quick Check:

   Use 'with TaskGroup(...) as ...:' [OK]

Hint: Use 'with TaskGroup("name") as var:' to create groups [OK]

Common Mistakes:

• Omitting 'with' keyword
• Using wrong capitalization like 'task_group'
• Not using 'as' to assign the group

3. Given the code snippet:

with TaskGroup('group1') as tg1:
    task_a = DummyOperator(task_id='a')
    task_b = DummyOperator(task_id='b')
task_c = DummyOperator(task_id='c')
task_a >> task_c

What will be the visual relationship between tasks in the Airflow UI?

medium

A. Tasks 'a', 'b', and 'c' are all grouped together

B. Tasks 'a' and 'b' are grouped; 'a' runs before 'c'; 'b' is independent

C. Task 'c' runs before 'a' and 'b' inside the group

D. Tasks 'a' and 'b' run after 'c'

Solution

1. Step 1: Analyze TaskGroup and tasks

   Tasks 'a' and 'b' are inside TaskGroup 'group1', so they appear grouped visually. Task 'c' is outside the group.

2. Step 2: Check dependencies

   Only 'task_a' has a downstream dependency to 'task_c', so 'a' runs before 'c'. 'b' has no dependencies and is independent.

3. Final Answer:

   Tasks 'a' and 'b' are grouped; 'a' runs before 'c'; 'b' is independent -> Option B

4. Quick Check:

   Group contains a,b; a >> c; b independent [OK]

Hint: Only tasks inside TaskGroup are grouped visually [OK]

Common Mistakes:

• Assuming all tasks are grouped
• Thinking 'b' depends on 'c'
• Confusing execution order

4. Identify the error in this Airflow DAG snippet using TaskGroup:

with TaskGroup('group1'):
    task1 = DummyOperator(task_id='task1')
task2 = DummyOperator(task_id='task2')
tg >> task2

medium

A. DummyOperator cannot be used inside TaskGroup

B. TaskGroup name must be uppercase

C. Task dependencies cannot cross TaskGroup boundaries

D. Missing 'as' keyword to assign TaskGroup to a variable

Solution

1. Step 1: Check TaskGroup usage

   The 'with TaskGroup('group1'):' statement should assign the group to a variable using 'as', e.g. 'with TaskGroup('group1') as tg:'

2. Step 2: Validate other options

   TaskGroup names can be lowercase. DummyOperator is allowed inside TaskGroup. Dependencies can cross groups.

3. Final Answer:

   Missing 'as' keyword to assign TaskGroup to a variable -> Option D

4. Quick Check:

   Use 'with TaskGroup(...) as var:' [OK]

Hint: Always use 'as' to assign TaskGroup in 'with' statement [OK]

Common Mistakes:

• Omitting 'as' in 'with TaskGroup(...) as ...:'
• Thinking TaskGroup names must be uppercase
• Believing DummyOperator is disallowed inside groups

5. You want to create a DAG with two task groups: 'extract' and 'transform'. Tasks in 'extract' must run before any in 'transform'. Which code snippet correctly sets this up?

hard

A. with TaskGroup('extract') as extract:
t1 = DummyOperator(task_id='t1')
t1b = DummyOperator(task_id='t1b')
with TaskGroup('transform') as transform:
t2 = DummyOperator(task_id='t2')
t2b = DummyOperator(task_id='t2b')
t1 >> t2

B. extract = TaskGroup('extract')
transform = TaskGroup('transform')
extract >> transform

C. with TaskGroup('extract') as extract:
t1 = DummyOperator(task_id='t1')
t1b = DummyOperator(task_id='t1b')
with TaskGroup('transform') as transform:
t2 = DummyOperator(task_id='t2')
t2b = DummyOperator(task_id='t2b')
extract >> transform

D. with TaskGroup('extract'):
t1 = DummyOperator(task_id='t1')
t1b = DummyOperator(task_id='t1b')
with TaskGroup('transform'):
t2 = DummyOperator(task_id='t2')
t2b = DummyOperator(task_id='t2b')
t1 >> t2

Solution

1. Step 1: Assign TaskGroups to variables

   Use 'with TaskGroup(...) as var:' to create groups and assign them to variables for referencing in dependencies.

2. Step 2: Set group-level dependencies

   Use extract >> transform to ensure every task in 'extract' (t1, t1b) completes before any task in 'transform' (t2, t2b) starts.

3. Step 3: Check options

   with TaskGroup('extract') as extract: t1 = ... t1b... with TaskGroup('transform') as transform: t2 = ... t2b... t1 >> t2 only links t1 to t2; t1b independent of transform tasks. extract = TaskGroup('extract') transform = TaskGroup('transform') extract >> transform misses 'with' context; no tasks added to groups. with TaskGroup('extract'): t1 = ... t1b... with TaskGroup('transform'): t2 = ... t2b... t1 >> t2 misses group variables and only links specific tasks. with TaskGroup('extract') as extract: t1 = ... t1b... with TaskGroup('transform') as transform: t2 = ... t2b... extract >> transform correctly uses group dependency.

4. Final Answer:

   with TaskGroup('extract') as extract: t1 = DummyOperator(task_id='t1') t1b = DummyOperator(task_id='t1b') with TaskGroup('transform') as transform: t2 = DummyOperator(task_id='t2') t2b = DummyOperator(task_id='t2b') extract >> transform -> Option C

5. Quick Check:

   tg1 >> tg2 for complete group ordering [OK]

Hint: Assign 'with TaskGroup(name) as tg:' then tg1 >> tg2 [OK]

Common Mistakes:

• Using individual task deps (t1 >> t2) with multiple tasks per group
• Missing 'with' context manager
• Omitting 'as var' when using group dependencies
• No tasks added without context manager