Task dependencies (>> and << operators) in Apache Airflow - Deep Dive

Overview - Task dependencies (>> and << operators)

What is it?

In Apache Airflow, task dependencies define the order in which tasks run. The operators >> and << are shortcuts to set these dependencies easily. Using >> means the task on the left runs before the task on the right. Using << means the task on the left runs after the task on the right. This helps organize workflows clearly and simply.

Why it matters

Without clear task dependencies, tasks might run in the wrong order or at the same time when they shouldn't. This can cause errors or incorrect results in data pipelines. The >> and << operators make it easy to set and read these dependencies, reducing mistakes and saving time. They help teams build reliable workflows that run smoothly.

Where it fits

Before learning task dependencies, you should understand what tasks and DAGs (Directed Acyclic Graphs) are in Airflow. After mastering dependencies, you can learn about advanced scheduling, branching, and error handling in workflows.

Mental Model

Core Idea

Task dependencies in Airflow define the order tasks run, and the >> and << operators are simple arrows showing which task comes before or after.

Think of it like...

It's like a relay race where runners pass the baton in order. The >> operator shows who passes the baton to whom, ensuring the race flows smoothly.

TaskA >> TaskB >> TaskC

Means:
TaskA runs first
↓
TaskB runs second
↓
TaskC runs last

Build-Up - 7 Steps

1

Foundation

Understanding Airflow Tasks and DAGs

Concept: Learn what tasks and DAGs are in Airflow and how they form workflows.

In Airflow, a task is a single unit of work, like running a script or copying a file. A DAG is a collection of tasks with defined order, forming a workflow. Each task runs once per DAG run. DAGs help organize and schedule tasks.

Result

You can identify tasks and understand that they need to be connected to form a workflow.

Knowing tasks and DAGs is essential because dependencies only make sense when you know what units you are connecting.

2

Foundation

Why Task Dependencies Matter

Concept: Understand why tasks need to run in a specific order and what happens without dependencies.

Some tasks depend on results from others. For example, you must extract data before transforming it. Without dependencies, Airflow might run tasks in any order or in parallel, causing errors or incomplete data.

Result

You see that dependencies prevent mistakes and ensure correct workflow execution.

Understanding the need for order helps you appreciate why dependency operators exist.

3

Intermediate

Using >> Operator for Dependencies

🤔Before reading on: do you think TaskA >> TaskB means TaskA runs before or after TaskB? Commit to your answer.

Concept: Learn how the >> operator sets a task to run before another task.

In Airflow, writing TaskA >> TaskB means TaskA must finish before TaskB starts. This is a simple way to set dependencies without calling set_upstream or set_downstream methods.

Result

TaskB will wait for TaskA to complete before running.

Knowing >> is a clear, readable way to set order makes workflow code easier to write and understand.

4

Intermediate

Using << Operator for Dependencies

🤔Before reading on: does TaskA << TaskB mean TaskA runs before or after TaskB? Commit to your answer.

Concept: Learn how the << operator sets a task to run after another task.

Writing TaskA << TaskB means TaskA runs after TaskB finishes. It is the reverse of >> but achieves the same dependency setup. This operator is useful when you want to express dependencies from the opposite direction.

Result

TaskA will wait for TaskB to complete before running.

Understanding << helps you read and write dependencies flexibly depending on which task you want to emphasize.

5

Intermediate

Chaining Multiple Tasks with >> and <<

🤔Before reading on: can you chain multiple tasks like TaskA >> TaskB >> TaskC? What order will they run in? Commit to your answer.

Concept: Learn how to chain several tasks in sequence using these operators.

You can write TaskA >> TaskB >> TaskC to mean TaskA runs first, then TaskB, then TaskC. Similarly, TaskC << TaskB << TaskA means the same order. This chaining makes defining long sequences simple and readable.

Result

Tasks run in the order TaskA, then TaskB, then TaskC.

Chaining operators lets you build clear, linear workflows without verbose code.

6

Advanced

Setting Parallel and Branching Dependencies

🤔Before reading on: if you write TaskA >> [TaskB, TaskC], do TaskB and TaskC run in order or parallel? Commit to your answer.

Concept: Learn how to set one task to run before multiple tasks and how tasks can run in parallel.

Using TaskA >> [TaskB, TaskC] means TaskA runs first, then both TaskB and TaskC run after TaskA finishes. TaskB and TaskC can run at the same time (parallel). This helps build branching workflows.

Result

TaskA runs first; TaskB and TaskC run next, possibly in parallel.

Knowing how to express parallelism with these operators helps design efficient workflows that save time.

7

Expert

How >> and << Operators Work Internally

🤔Before reading on: do you think >> and << create new tasks or just link existing ones? Commit to your answer.

Concept: Understand the internal mechanism of these operators in Airflow's task dependency graph.

The >> and << operators do not create tasks. They call methods on existing task objects to update their upstream and downstream sets. This modifies the DAG's dependency graph. Internally, they use Python's operator overloading to make syntax simple but actually call set_downstream or set_upstream methods.

Result

Dependencies are updated in the DAG's graph structure without creating new tasks.

Knowing this prevents confusion about task creation and helps debug dependency issues by understanding the underlying graph updates.

Under the Hood

The >> and << operators are Python methods overloaded on Airflow's BaseOperator class. When you write TaskA >> TaskB, it calls TaskA.set_downstream(TaskB), adding TaskB to TaskA's downstream tasks and TaskA to TaskB's upstream tasks. This updates the DAG's directed acyclic graph structure that Airflow uses to schedule tasks in order. The DAG object keeps track of all tasks and their dependencies to ensure correct execution order.

Why designed this way?

Airflow's designers wanted a simple, readable way to define dependencies without verbose method calls. Using Python's operator overloading for >> and << provides an intuitive arrow-like syntax that matches the concept of task order. This design balances code clarity with powerful graph management. Alternatives like only using set_upstream/set_downstream were more verbose and less readable.

┌─────────┐     >>     ┌─────────┐
│ Task A  │──────────▶│ Task B  │
└─────────┘           └─────────┘

Internally:
TaskA.set_downstream(TaskB)
TaskB.set_upstream(TaskA)

DAG Graph:
TaskA --> TaskB

Myth Busters - 4 Common Misconceptions

Quick: Does TaskA >> TaskB mean TaskB runs before TaskA? Commit yes or no.

Common Belief: Some think TaskA >> TaskB means TaskB runs before TaskA.

Reality: TaskA >> TaskB means TaskA runs first, then TaskB runs after.

Why it matters: Misunderstanding this reverses task order, causing workflows to fail or produce wrong results.

Quick: Does TaskA >> TaskB create a new task? Commit yes or no.

Common Belief: Some believe using >> creates new tasks automatically.

Reality: >> only links existing tasks; it does not create any new tasks.

Why it matters: Thinking it creates tasks can lead to duplicate tasks or confusion about workflow structure.

Quick: If you write TaskA >> [TaskB, TaskC], do TaskB and TaskC run one after another or in parallel? Commit your answer.

Common Belief: Some assume TaskB and TaskC run sequentially after TaskA.

Reality: TaskB and TaskC run in parallel after TaskA finishes.

Why it matters: Misunderstanding this can cause incorrect assumptions about workflow timing and resource use.

Quick: Does TaskA << TaskB mean the same as TaskA >> TaskB? Commit yes or no.

Common Belief: Some think << and >> are interchangeable and mean the same.

Reality: TaskA << TaskB means TaskA runs after TaskB, which is the reverse of TaskA >> TaskB.

Why it matters: Confusing these operators leads to reversed dependencies and broken workflows.

Expert Zone

1

The >> and << operators return the right-hand task, allowing chaining of dependencies in a single expression.

2

Using lists with >> or << creates multiple dependencies efficiently but can lead to complex graphs if overused without clear structure.

3

Dependencies set by >> and << are stored in the DAG's internal graph, which can be inspected or modified programmatically for dynamic workflows.

When NOT to use

For very complex or dynamic dependencies, using explicit set_upstream and set_downstream calls or the TaskGroup feature may be clearer. Also, for conditional branching, Airflow's BranchPythonOperator is better suited than just chaining with >> or <<.

Production Patterns

In production, teams use >> and << to define clear linear and parallel task flows. They combine these operators with TaskGroups for modular workflows and use dynamic DAG generation to handle variable dependencies. Proper use of these operators ensures maintainable, readable, and scalable pipelines.

Connections

Directed Acyclic Graphs (DAGs)

Task dependencies form the edges of a DAG, defining task order.

Understanding DAGs helps grasp why dependencies must not form cycles and how Airflow schedules tasks.

Event-driven Programming

Task dependencies resemble event triggers where one task's completion triggers the next.

Knowing event-driven patterns clarifies how Airflow manages task execution flow based on dependencies.

Project Management Critical Path

Task dependencies in Airflow are like task sequences in project management critical path analysis.

Recognizing this connection helps understand the importance of order and parallelism in workflows.

Common Pitfalls

#1 Reversing dependency order and causing wrong execution sequence.

Wrong approach:

task_b >> task_a

Correct approach:

task_a >> task_b

Root cause:

Misunderstanding that >> means left task runs before right task.

#2 Assuming >> creates new tasks instead of linking existing ones.

Wrong approach:

task_a = DummyOperator(task_id='a')
task_b = task_a >> DummyOperator(task_id='b')

Correct approach:

task_a = DummyOperator(task_id='a')
task_b = DummyOperator(task_id='b')
task_a >> task_b

Root cause:

Confusing dependency operators with task creation syntax.

#3 Using >> with a list but expecting sequential execution of all tasks in the list.

Wrong approach:

task_a >> [task_b, task_c] # expecting task_b then task_c

Correct approach:

task_a >> task_b >> task_c # for sequential execution

Root cause:

Not realizing that list syntax runs tasks in parallel, not sequence.

Key Takeaways

The >> and << operators in Airflow set task dependencies clearly and simply, defining execution order.

>> means the task on the left runs before the task on the right; << means the opposite.

Chaining these operators lets you build linear or parallel workflows with readable code.

These operators modify the DAG's internal graph by linking existing tasks, not creating new ones.

Understanding these operators prevents common mistakes that break workflows or cause unexpected task order.

Practice

1. In Airflow, what does the operator task1 >> task2 mean?

easy

A. task1 runs before task2

B. task1 runs after task2

C. task1 and task2 run at the same time

D. task1 is unrelated to task2

Solution

1. Step 1: Understand the meaning of >> operator

   The >> operator in Airflow means "runs before" or "upstream to downstream" dependency.

2. Step 2: Apply to given tasks

   Given task1 >> task2, task1 must finish before task2 starts.

3. Final Answer:

   task1 runs before task2 -> Option A

4. Quick Check:

   >> means runs before [OK]

Hint: Remember: >> means runs before, << means runs after [OK]

Common Mistakes:

• Confusing >> with <<
• Thinking >> means runs after
• Assuming tasks run in parallel

2. Which of the following is the correct syntax to set task2 to run after task1 using Airflow operators?

easy

A. task1 >> task2

B. task2 << task1

C. task2 >> task1

D. task1 << task2

Solution

1. Step 1: Identify the operator meaning

   >> means "runs before"; so task1 >> task2 means task1 runs before task2.

2. Step 2: Match the requirement

   To make task2 run after task1, task1 must run before task2, so task1 >> task2 is correct.

3. Final Answer:

   task1 >> task2 -> Option A

4. Quick Check:

   task2 after task1 = task1 >> task2 [OK]

Hint: Use >> to say runs before, so task1 >> task2 means task2 after task1 [OK]

Common Mistakes:

• Using << backwards
• Swapping task order
• Confusing which task is upstream

3. Given the code snippet:

task1 >> task2 << task3

Which statement is true about the task execution order?

medium

A. task2 runs before task1 and task3

B. task1 runs after task2, and task3 runs after task2

C. task1 runs before task2, and task3 runs before task2

D. task1, task2, and task3 run in parallel

Solution

1. Step 1: Break down the dependencies

   task1 >> task2 means task1 runs before task2; task3 << task2 means task3 runs before task2 (because << means runs after, so task2 runs after task3).

2. Step 2: Combine the relations

   Both task1 and task3 must finish before task2 starts.

3. Final Answer:

   task1 runs before task2, and task3 runs before task2 -> Option C

4. Quick Check:

   task1 and task3 before task2 = task1 runs before task2, and task3 runs before task2 [OK]

Hint: >> means runs before; << means runs after (reverse order) [OK]

Common Mistakes:

• Misreading << as runs before
• Assuming task2 runs first
• Thinking tasks run in parallel

4. Identify the error in this Airflow task dependency code:

task1 >> >> task2

medium

A. task2 runs before task1

B. Syntax error due to double >> operators

C. No error, tasks run sequentially

D. task1 and task2 run in parallel

Solution

1. Step 1: Analyze the operator usage

   The code has two consecutive >> operators without a task in between, which is invalid syntax in Python and Airflow.

2. Step 2: Understand Airflow dependency syntax

   Only one >> operator is allowed between two tasks to set dependency; double >> causes syntax error.

3. Final Answer:

   Syntax error due to double >> operators -> Option B

4. Quick Check:

   Double >> is invalid syntax [OK]

Hint: Only one >> or << between tasks, no doubles [OK]

Common Mistakes:

• Writing multiple >> operators in a row
• Ignoring syntax errors
• Assuming double >> means stronger dependency

5. You want to create a DAG where task1 and task3 run first in parallel, then task2 runs after both finish. Which code correctly sets this dependency?

hard

A. task2 >> [task1, task3]

B. task1 << task2 >> task3

C. task1 >> task2 >> task3

D. [task1, task3] >> task2

Solution

1. Step 1: Understand the requirement

   task1 and task3 run in parallel first, then task2 runs after both finish.

2. Step 2: Translate to Airflow syntax

   Using list syntax [task1, task3] >> task2 means both task1 and task3 run before task2.

3. Step 3: Check other options

   task1 >> task2 >> task3 runs tasks sequentially; B and D have wrong directions.

4. Final Answer:

   [task1, task3] >> task2 -> Option D

5. Quick Check:

   List >> task means all list tasks before next [OK]

Hint: Use list syntax [task1, task3] >> task2 for parallel upstream tasks [OK]

Common Mistakes:

• Using >> in wrong order
• Not using list for parallel tasks
• Confusing << with >>