Task dependencies (>> and << operators) in Apache Airflow - Step-by-Step Execution

Process Flow - Task dependencies (>> and << operators)

Define Task A

↓

Set Dependency

↓

Execution Order

↓

Task Runs in Order

This flow shows how two tasks are defined and linked using >> or << operators to set execution order.

Execution Sample

Apache Airflow

task1 >> task2
# or
# task2 << task1

This code sets task1 to run before task2 using the >> operator (or equivalently << operator).

Process Table

Step	Action	Operator	Dependency Set	Resulting Order
1	Define task1	N/A	None	No order yet
2	Define task2	N/A	None	No order yet
3	Set dependency task1 >> task2	>>	task1 before task2	task1 runs before task2
4	Check dependency task2 << task1	<<	task1 before task2	task1 runs before task2
5	Execute tasks	N/A	task1 before task2	task1 runs first, then task2
6	End	N/A	N/A	Execution complete

💡 Execution stops after tasks run in the defined order.

Status Tracker

Variable	Start	After Step 3	After Step 5	Final
task1	Defined	Dependency set as upstream	Executed first	Completed
task2	Defined	Dependency set as downstream	Executed second	Completed

Key Moments - 3 Insights

Why does task1 >> task2 mean task1 runs before task2?

Because the >> operator sets the left task as upstream, so task2 waits for task1 to finish (see execution_table step 3).

Is task2 << task1 the same as task1 >> task2?

Yes, << sets the right task as downstream, so task2 << task1 means task1 runs before task2 (see execution_table step 4).

What happens if no dependencies are set between tasks?

Tasks run independently and may execute in any order or in parallel (see execution_table steps 1 and 2).

Visual Quiz - 3 Questions

Look at the execution table, what dependency is set at step 3?

A No dependency set

B task2 runs before task1

C task1 runs before task2

D Tasks run in parallel

💡 Hint

Check the 'Dependency Set' column at step 3 in the execution_table.

At which step do tasks actually execute in order?

A Step 2

B Step 5

C Step 3

D Step 6

💡 Hint

Look for 'Execute tasks' action in the execution_table.

If we remove the dependency operator, what changes in the execution?

A Tasks run independently or in parallel

B Tasks run in defined order

C Tasks fail to run

D Tasks run twice

💡 Hint

Refer to key_moments about no dependencies set (execution_table steps 1 and 2).

Concept Snapshot

Task dependencies in Airflow use >> and << operators.
'task1 >> task2' means task1 runs before task2.
'task2 << task1' means the same.
Without dependencies, tasks run independently.
Use these operators to control execution order.

Full Transcript

In Airflow, you define tasks and set their execution order using the >> and << operators. For example, 'task1 >> task2' means task1 must finish before task2 starts. The >> operator sets the left task as upstream, so the right task waits. The << operator works the opposite way but achieves the same result. If no dependencies are set, tasks run independently and may execute in any order or in parallel. This visual trace showed defining tasks, setting dependencies, and the resulting execution order step-by-step.

Practice

1. In Airflow, what does the operator task1 >> task2 mean?

easy

A. task1 runs before task2

B. task1 runs after task2

C. task1 and task2 run at the same time

D. task1 is unrelated to task2

Solution

1. Step 1: Understand the meaning of >> operator

   The >> operator in Airflow means "runs before" or "upstream to downstream" dependency.

2. Step 2: Apply to given tasks

   Given task1 >> task2, task1 must finish before task2 starts.

3. Final Answer:

   task1 runs before task2 -> Option A

4. Quick Check:

   >> means runs before [OK]

Hint: Remember: >> means runs before, << means runs after [OK]

Common Mistakes:

• Confusing >> with <<
• Thinking >> means runs after
• Assuming tasks run in parallel

2. Which of the following is the correct syntax to set task2 to run after task1 using Airflow operators?

easy

A. task1 >> task2

B. task2 << task1

C. task2 >> task1

D. task1 << task2

Solution

1. Step 1: Identify the operator meaning

   >> means "runs before"; so task1 >> task2 means task1 runs before task2.

2. Step 2: Match the requirement

   To make task2 run after task1, task1 must run before task2, so task1 >> task2 is correct.

3. Final Answer:

   task1 >> task2 -> Option A

4. Quick Check:

   task2 after task1 = task1 >> task2 [OK]

Hint: Use >> to say runs before, so task1 >> task2 means task2 after task1 [OK]

Common Mistakes:

• Using << backwards
• Swapping task order
• Confusing which task is upstream

3. Given the code snippet:

task1 >> task2 << task3

Which statement is true about the task execution order?

medium

A. task2 runs before task1 and task3

B. task1 runs after task2, and task3 runs after task2

C. task1 runs before task2, and task3 runs before task2

D. task1, task2, and task3 run in parallel

Solution

1. Step 1: Break down the dependencies

   task1 >> task2 means task1 runs before task2; task3 << task2 means task3 runs before task2 (because << means runs after, so task2 runs after task3).

2. Step 2: Combine the relations

   Both task1 and task3 must finish before task2 starts.

3. Final Answer:

   task1 runs before task2, and task3 runs before task2 -> Option C

4. Quick Check:

   task1 and task3 before task2 = task1 runs before task2, and task3 runs before task2 [OK]

Hint: >> means runs before; << means runs after (reverse order) [OK]

Common Mistakes:

• Misreading << as runs before
• Assuming task2 runs first
• Thinking tasks run in parallel

4. Identify the error in this Airflow task dependency code:

task1 >> >> task2

medium

A. task2 runs before task1

B. Syntax error due to double >> operators

C. No error, tasks run sequentially

D. task1 and task2 run in parallel

Solution

1. Step 1: Analyze the operator usage

   The code has two consecutive >> operators without a task in between, which is invalid syntax in Python and Airflow.

2. Step 2: Understand Airflow dependency syntax

   Only one >> operator is allowed between two tasks to set dependency; double >> causes syntax error.

3. Final Answer:

   Syntax error due to double >> operators -> Option B

4. Quick Check:

   Double >> is invalid syntax [OK]

Hint: Only one >> or << between tasks, no doubles [OK]

Common Mistakes:

• Writing multiple >> operators in a row
• Ignoring syntax errors
• Assuming double >> means stronger dependency

5. You want to create a DAG where task1 and task3 run first in parallel, then task2 runs after both finish. Which code correctly sets this dependency?

hard

A. task2 >> [task1, task3]

B. task1 << task2 >> task3

C. task1 >> task2 >> task3

D. [task1, task3] >> task2

Solution

1. Step 1: Understand the requirement

   task1 and task3 run in parallel first, then task2 runs after both finish.

2. Step 2: Translate to Airflow syntax

   Using list syntax [task1, task3] >> task2 means both task1 and task3 run before task2.

3. Step 3: Check other options

   task1 >> task2 >> task3 runs tasks sequentially; B and D have wrong directions.

4. Final Answer:

   [task1, task3] >> task2 -> Option D

5. Quick Check:

   List >> task means all list tasks before next [OK]

Hint: Use list syntax [task1, task3] >> task2 for parallel upstream tasks [OK]

Common Mistakes:

• Using >> in wrong order
• Not using list for parallel tasks
• Confusing << with >>