Task dependencies (>> and << operators) in Apache Airflow - Commands & Configuration

Introduction

When you have multiple tasks in Airflow, you need to control the order they run. Task dependencies let you say which task should run before or after another. The >> and << operators make it easy to set these orders in a simple way.

When you want task B to start only after task A finishes successfully.

When you have a sequence of tasks that must run one after another.

When you want to run multiple tasks in parallel after one task completes.

When you want to define complex workflows with clear task order.

When you want to improve readability of your DAG by using simple operators for dependencies.

Config File - example_dag.py

example_dag.py

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime

default_args = {
    'start_date': datetime(2024, 1, 1),
}

dag = DAG('example_task_dependencies', default_args=default_args, schedule_interval='@daily')

# Define tasks
start = BashOperator(task_id='start', bash_command='echo Start', dag=dag)
task1 = BashOperator(task_id='task1', bash_command='echo Task 1', dag=dag)
task2 = BashOperator(task_id='task2', bash_command='echo Task 2', dag=dag)
end = BashOperator(task_id='end', bash_command='echo End', dag=dag)

# Set dependencies using >> and << operators
start >> task1 >> task2 >> end

# Alternative: end << task2 << task1 << start

This DAG file defines four tasks: start, task1, task2, and end.

The >> operator sets the order so that start runs before task1, which runs before task2, which runs before end.

The alternative line shows the same order using the << operator, which means the task on the left depends on the task on the right.

Commands

List all available DAGs to confirm your DAG file is recognized by Airflow.

Terminal

airflow dags list

Expected Output

example_task_dependencies

List all tasks in the DAG to verify the tasks are defined correctly.

Terminal

airflow tasks list example_task_dependencies

Expected Output

start

task1

task2

end

Run the 'start' task manually for the given date to check it executes without errors.

Terminal

airflow tasks test example_task_dependencies start 2024-01-01

Expected Output

[2024-01-01 00:00:00,000] {bash.py:123} INFO - Running command: echo Start

[2024-01-01 00:00:00,100] {bash.py:130} INFO - Output:

Start

[2024-01-01 00:00:00,200] {taskinstance.py:123} INFO - Task succeeded

Run 'task1' manually to verify it runs after 'start' in the dependency chain.

Terminal

airflow tasks test example_task_dependencies task1 2024-01-01

Expected Output

[2024-01-01 00:00:01,000] {bash.py:123} INFO - Running command: echo Task 1

[2024-01-01 00:00:01,100] {bash.py:130} INFO - Output:

Task 1

[2024-01-01 00:00:01,200] {taskinstance.py:123} INFO - Task succeeded

Key Concept

If you remember nothing else from this pattern, remember: the >> operator means 'runs before' and the << operator means 'runs after' to set task order simply.

Common Mistakes

Using >> or << operators without assigning tasks to variables first

You cannot set dependencies on undefined tasks; this causes errors when parsing the DAG.

Always define tasks as variables before using >> or << to link them.

Confusing the direction of >> and << operators

This leads to wrong task order and unexpected workflow behavior.

Remember: A >> B means A runs before B; A << B means A runs after B.

Not chaining dependencies properly, mixing >> and << in confusing ways

This can create unclear or incorrect task sequences.

Use one style consistently and chain tasks clearly for readability.

Summary

Define tasks as variables before setting dependencies.

Use >> to say one task runs before another.

Use << to say one task runs after another.

Chain tasks with these operators to create clear workflows.

Practice

1. In Airflow, what does the operator task1 >> task2 mean?

easy

A. task1 runs before task2

B. task1 runs after task2

C. task1 and task2 run at the same time

D. task1 is unrelated to task2

Solution

1. Step 1: Understand the meaning of >> operator

   The >> operator in Airflow means "runs before" or "upstream to downstream" dependency.

2. Step 2: Apply to given tasks

   Given task1 >> task2, task1 must finish before task2 starts.

3. Final Answer:

   task1 runs before task2 -> Option A

4. Quick Check:

   >> means runs before [OK]

Hint: Remember: >> means runs before, << means runs after [OK]

Common Mistakes:

• Confusing >> with <<
• Thinking >> means runs after
• Assuming tasks run in parallel

2. Which of the following is the correct syntax to set task2 to run after task1 using Airflow operators?

easy

A. task1 >> task2

B. task2 << task1

C. task2 >> task1

D. task1 << task2

Solution

1. Step 1: Identify the operator meaning

   >> means "runs before"; so task1 >> task2 means task1 runs before task2.

2. Step 2: Match the requirement

   To make task2 run after task1, task1 must run before task2, so task1 >> task2 is correct.

3. Final Answer:

   task1 >> task2 -> Option A

4. Quick Check:

   task2 after task1 = task1 >> task2 [OK]

Hint: Use >> to say runs before, so task1 >> task2 means task2 after task1 [OK]

Common Mistakes:

• Using << backwards
• Swapping task order
• Confusing which task is upstream

3. Given the code snippet:

task1 >> task2 << task3

Which statement is true about the task execution order?

medium

A. task2 runs before task1 and task3

B. task1 runs after task2, and task3 runs after task2

C. task1 runs before task2, and task3 runs before task2

D. task1, task2, and task3 run in parallel

Solution

1. Step 1: Break down the dependencies

   task1 >> task2 means task1 runs before task2; task3 << task2 means task3 runs before task2 (because << means runs after, so task2 runs after task3).

2. Step 2: Combine the relations

   Both task1 and task3 must finish before task2 starts.

3. Final Answer:

   task1 runs before task2, and task3 runs before task2 -> Option C

4. Quick Check:

   task1 and task3 before task2 = task1 runs before task2, and task3 runs before task2 [OK]

Hint: >> means runs before; << means runs after (reverse order) [OK]

Common Mistakes:

• Misreading << as runs before
• Assuming task2 runs first
• Thinking tasks run in parallel

4. Identify the error in this Airflow task dependency code:

task1 >> >> task2

medium

A. task2 runs before task1

B. Syntax error due to double >> operators

C. No error, tasks run sequentially

D. task1 and task2 run in parallel

Solution

1. Step 1: Analyze the operator usage

   The code has two consecutive >> operators without a task in between, which is invalid syntax in Python and Airflow.

2. Step 2: Understand Airflow dependency syntax

   Only one >> operator is allowed between two tasks to set dependency; double >> causes syntax error.

3. Final Answer:

   Syntax error due to double >> operators -> Option B

4. Quick Check:

   Double >> is invalid syntax [OK]

Hint: Only one >> or << between tasks, no doubles [OK]

Common Mistakes:

• Writing multiple >> operators in a row
• Ignoring syntax errors
• Assuming double >> means stronger dependency

5. You want to create a DAG where task1 and task3 run first in parallel, then task2 runs after both finish. Which code correctly sets this dependency?

hard

A. task2 >> [task1, task3]

B. task1 << task2 >> task3

C. task1 >> task2 >> task3

D. [task1, task3] >> task2

Solution

1. Step 1: Understand the requirement

   task1 and task3 run in parallel first, then task2 runs after both finish.

2. Step 2: Translate to Airflow syntax

   Using list syntax [task1, task3] >> task2 means both task1 and task3 run before task2.

3. Step 3: Check other options

   task1 >> task2 >> task3 runs tasks sequentially; B and D have wrong directions.

4. Final Answer:

   [task1, task3] >> task2 -> Option D

5. Quick Check:

   List >> task means all list tasks before next [OK]

Hint: Use list syntax [task1, task3] >> task2 for parallel upstream tasks [OK]

Common Mistakes:

• Using >> in wrong order
• Not using list for parallel tasks
• Confusing << with >>