Task dependencies (>> and << operators) in Apache Airflow - Deep Dive

Overview - Task dependencies (>> and << operators)

What is it?

In Apache Airflow, task dependencies define the order in which tasks run. The operators >> and << are shortcuts to set these dependencies easily. Using >> means the task on the left runs before the task on the right. Using << means the task on the left runs after the task on the right. This helps organize workflows clearly and simply.

Why it matters

Without clear task dependencies, tasks might run in the wrong order or at the same time when they shouldn't. This can cause errors or incorrect results in data pipelines. The >> and << operators make it easy to set and read these dependencies, reducing mistakes and saving time. They help teams build reliable workflows that run smoothly.

Where it fits

Before learning task dependencies, you should understand what tasks and DAGs (Directed Acyclic Graphs) are in Airflow. After mastering dependencies, you can learn about advanced scheduling, branching, and error handling in workflows.

Mental Model

Core Idea

Task dependencies in Airflow define the order tasks run, and the >> and << operators are simple arrows showing which task comes before or after.

Think of it like...

It's like a relay race where runners pass the baton in order. The >> operator shows who passes the baton to whom, ensuring the race flows smoothly.

TaskA >> TaskB >> TaskC

Means:
TaskA runs first
↓
TaskB runs second
↓
TaskC runs last

Build-Up - 7 Steps

1

Foundation

Understanding Airflow Tasks and DAGs

Concept: Learn what tasks and DAGs are in Airflow and how they form workflows.

In Airflow, a task is a single unit of work, like running a script or copying a file. A DAG is a collection of tasks with defined order, forming a workflow. Each task runs once per DAG run. DAGs help organize and schedule tasks.

Result

You can identify tasks and understand that they need to be connected to form a workflow.

Knowing tasks and DAGs is essential because dependencies only make sense when you know what units you are connecting.

2

Foundation

Why Task Dependencies Matter

Concept: Understand why tasks need to run in a specific order and what happens without dependencies.

Some tasks depend on results from others. For example, you must extract data before transforming it. Without dependencies, Airflow might run tasks in any order or in parallel, causing errors or incomplete data.

Result

You see that dependencies prevent mistakes and ensure correct workflow execution.

Understanding the need for order helps you appreciate why dependency operators exist.

3

Intermediate

Using >> Operator for Dependencies

🤔Before reading on: do you think TaskA >> TaskB means TaskA runs before or after TaskB? Commit to your answer.

Concept: Learn how the >> operator sets a task to run before another task.

In Airflow, writing TaskA >> TaskB means TaskA must finish before TaskB starts. This is a simple way to set dependencies without calling set_upstream or set_downstream methods.

Result

TaskB will wait for TaskA to complete before running.

Knowing >> is a clear, readable way to set order makes workflow code easier to write and understand.

4

Intermediate

Using << Operator for Dependencies

🤔Before reading on: does TaskA << TaskB mean TaskA runs before or after TaskB? Commit to your answer.

Concept: Learn how the << operator sets a task to run after another task.

Writing TaskA << TaskB means TaskA runs after TaskB finishes. It is the reverse of >> but achieves the same dependency setup. This operator is useful when you want to express dependencies from the opposite direction.

Result

TaskA will wait for TaskB to complete before running.

Understanding << helps you read and write dependencies flexibly depending on which task you want to emphasize.

5

Intermediate

Chaining Multiple Tasks with >> and <<

🤔Before reading on: can you chain multiple tasks like TaskA >> TaskB >> TaskC? What order will they run in? Commit to your answer.

Concept: Learn how to chain several tasks in sequence using these operators.

You can write TaskA >> TaskB >> TaskC to mean TaskA runs first, then TaskB, then TaskC. Similarly, TaskC << TaskB << TaskA means the same order. This chaining makes defining long sequences simple and readable.

Result

Tasks run in the order TaskA, then TaskB, then TaskC.

Chaining operators lets you build clear, linear workflows without verbose code.

6

Advanced

Setting Parallel and Branching Dependencies

🤔Before reading on: if you write TaskA >> [TaskB, TaskC], do TaskB and TaskC run in order or parallel? Commit to your answer.

Concept: Learn how to set one task to run before multiple tasks and how tasks can run in parallel.

Using TaskA >> [TaskB, TaskC] means TaskA runs first, then both TaskB and TaskC run after TaskA finishes. TaskB and TaskC can run at the same time (parallel). This helps build branching workflows.

Result

TaskA runs first; TaskB and TaskC run next, possibly in parallel.

Knowing how to express parallelism with these operators helps design efficient workflows that save time.

7

Expert

How >> and << Operators Work Internally

🤔Before reading on: do you think >> and << create new tasks or just link existing ones? Commit to your answer.

Concept: Understand the internal mechanism of these operators in Airflow's task dependency graph.

The >> and << operators do not create tasks. They call methods on existing task objects to update their upstream and downstream sets. This modifies the DAG's dependency graph. Internally, they use Python's operator overloading to make syntax simple but actually call set_downstream or set_upstream methods.

Result

Dependencies are updated in the DAG's graph structure without creating new tasks.

Knowing this prevents confusion about task creation and helps debug dependency issues by understanding the underlying graph updates.

Under the Hood

The >> and << operators are Python methods overloaded on Airflow's BaseOperator class. When you write TaskA >> TaskB, it calls TaskA.set_downstream(TaskB), adding TaskB to TaskA's downstream tasks and TaskA to TaskB's upstream tasks. This updates the DAG's directed acyclic graph structure that Airflow uses to schedule tasks in order. The DAG object keeps track of all tasks and their dependencies to ensure correct execution order.

Why designed this way?

Airflow's designers wanted a simple, readable way to define dependencies without verbose method calls. Using Python's operator overloading for >> and << provides an intuitive arrow-like syntax that matches the concept of task order. This design balances code clarity with powerful graph management. Alternatives like only using set_upstream/set_downstream were more verbose and less readable.

┌─────────┐     >>     ┌─────────┐
│ Task A  │──────────▶│ Task B  │
└─────────┘           └─────────┘

Internally:
TaskA.set_downstream(TaskB)
TaskB.set_upstream(TaskA)

DAG Graph:
TaskA --> TaskB

Myth Busters - 4 Common Misconceptions

Quick: Does TaskA >> TaskB mean TaskB runs before TaskA? Commit yes or no.

Common Belief: Some think TaskA >> TaskB means TaskB runs before TaskA.

Reality: TaskA >> TaskB means TaskA runs first, then TaskB runs after.

Why it matters: Misunderstanding this reverses task order, causing workflows to fail or produce wrong results.

Quick: Does TaskA >> TaskB create a new task? Commit yes or no.

Common Belief: Some believe using >> creates new tasks automatically.

Reality: >> only links existing tasks; it does not create any new tasks.

Why it matters: Thinking it creates tasks can lead to duplicate tasks or confusion about workflow structure.

Quick: If you write TaskA >> [TaskB, TaskC], do TaskB and TaskC run one after another or in parallel? Commit your answer.

Common Belief: Some assume TaskB and TaskC run sequentially after TaskA.

Reality: TaskB and TaskC run in parallel after TaskA finishes.

Why it matters: Misunderstanding this can cause incorrect assumptions about workflow timing and resource use.

Quick: Does TaskA << TaskB mean the same as TaskA >> TaskB? Commit yes or no.

Common Belief: Some think << and >> are interchangeable and mean the same.

Reality: TaskA << TaskB means TaskA runs after TaskB, which is the reverse of TaskA >> TaskB.

Why it matters: Confusing these operators leads to reversed dependencies and broken workflows.

Expert Zone

1

The >> and << operators return the right-hand task, allowing chaining of dependencies in a single expression.

2

Using lists with >> or << creates multiple dependencies efficiently but can lead to complex graphs if overused without clear structure.

3

Dependencies set by >> and << are stored in the DAG's internal graph, which can be inspected or modified programmatically for dynamic workflows.

When NOT to use

For very complex or dynamic dependencies, using explicit set_upstream and set_downstream calls or the TaskGroup feature may be clearer. Also, for conditional branching, Airflow's BranchPythonOperator is better suited than just chaining with >> or <<.

Production Patterns

In production, teams use >> and << to define clear linear and parallel task flows. They combine these operators with TaskGroups for modular workflows and use dynamic DAG generation to handle variable dependencies. Proper use of these operators ensures maintainable, readable, and scalable pipelines.

Connections

Directed Acyclic Graphs (DAGs)

Task dependencies form the edges of a DAG, defining task order.

Understanding DAGs helps grasp why dependencies must not form cycles and how Airflow schedules tasks.

Event-driven Programming

Task dependencies resemble event triggers where one task's completion triggers the next.

Knowing event-driven patterns clarifies how Airflow manages task execution flow based on dependencies.

Project Management Critical Path

Task dependencies in Airflow are like task sequences in project management critical path analysis.

Recognizing this connection helps understand the importance of order and parallelism in workflows.

Common Pitfalls

#1 Reversing dependency order and causing wrong execution sequence.

Wrong approach:

task_b >> task_a

Correct approach:

task_a >> task_b

Root cause:

Misunderstanding that >> means left task runs before right task.

#2 Assuming >> creates new tasks instead of linking existing ones.

Wrong approach:

task_a = DummyOperator(task_id='a')
task_b = task_a >> DummyOperator(task_id='b')

Correct approach:

task_a = DummyOperator(task_id='a')
task_b = DummyOperator(task_id='b')
task_a >> task_b

Root cause:

Confusing dependency operators with task creation syntax.

#3 Using >> with a list but expecting sequential execution of all tasks in the list.

Wrong approach:

task_a >> [task_b, task_c] # expecting task_b then task_c

Correct approach:

task_a >> task_b >> task_c # for sequential execution

Root cause:

Not realizing that list syntax runs tasks in parallel, not sequence.

Key Takeaways

The >> and << operators in Airflow set task dependencies clearly and simply, defining execution order.

>> means the task on the left runs before the task on the right; << means the opposite.

Chaining these operators lets you build linear or parallel workflows with readable code.

These operators modify the DAG's internal graph by linking existing tasks, not creating new ones.

Understanding these operators prevents common mistakes that break workflows or cause unexpected task order.