Task dependencies (>> and << operators) in Apache Airflow - Commands & Configuration

Introduction

When you have multiple tasks in Airflow, you need to control the order they run. Task dependencies let you say which task should run before or after another. The >> and << operators make it easy to set these orders in a simple way.

When you want task B to start only after task A finishes successfully.

When you have a sequence of tasks that must run one after another.

When you want to run multiple tasks in parallel after one task completes.

When you want to define complex workflows with clear task order.

When you want to improve readability of your DAG by using simple operators for dependencies.

Config File - example_dag.py

example_dag.py

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime

default_args = {
    'start_date': datetime(2024, 1, 1),
}

dag = DAG('example_task_dependencies', default_args=default_args, schedule_interval='@daily')

# Define tasks
start = BashOperator(task_id='start', bash_command='echo Start', dag=dag)
task1 = BashOperator(task_id='task1', bash_command='echo Task 1', dag=dag)
task2 = BashOperator(task_id='task2', bash_command='echo Task 2', dag=dag)
end = BashOperator(task_id='end', bash_command='echo End', dag=dag)

# Set dependencies using >> and << operators
start >> task1 >> task2 >> end

# Alternative: end << task2 << task1 << start

This DAG file defines four tasks: start, task1, task2, and end.

The >> operator sets the order so that start runs before task1, which runs before task2, which runs before end.

The alternative line shows the same order using the << operator, which means the task on the left depends on the task on the right.

Commands

List all available DAGs to confirm your DAG file is recognized by Airflow.

Terminal

airflow dags list

Expected Output

example_task_dependencies

List all tasks in the DAG to verify the tasks are defined correctly.

Terminal

airflow tasks list example_task_dependencies

Expected Output

start

task1

task2

end

Run the 'start' task manually for the given date to check it executes without errors.

Terminal

airflow tasks test example_task_dependencies start 2024-01-01

Expected Output

[2024-01-01 00:00:00,000] {bash.py:123} INFO - Running command: echo Start

[2024-01-01 00:00:00,100] {bash.py:130} INFO - Output:

Start

[2024-01-01 00:00:00,200] {taskinstance.py:123} INFO - Task succeeded

Run 'task1' manually to verify it runs after 'start' in the dependency chain.

Terminal

airflow tasks test example_task_dependencies task1 2024-01-01

Expected Output

[2024-01-01 00:00:01,000] {bash.py:123} INFO - Running command: echo Task 1

[2024-01-01 00:00:01,100] {bash.py:130} INFO - Output:

Task 1

[2024-01-01 00:00:01,200] {taskinstance.py:123} INFO - Task succeeded

Key Concept

If you remember nothing else from this pattern, remember: the >> operator means 'runs before' and the << operator means 'runs after' to set task order simply.

Common Mistakes

Using >> or << operators without assigning tasks to variables first

You cannot set dependencies on undefined tasks; this causes errors when parsing the DAG.

Always define tasks as variables before using >> or << to link them.

Confusing the direction of >> and << operators

This leads to wrong task order and unexpected workflow behavior.

Remember: A >> B means A runs before B; A << B means A runs after B.

Not chaining dependencies properly, mixing >> and << in confusing ways

This can create unclear or incorrect task sequences.

Use one style consistently and chain tasks clearly for readability.

Summary

Define tasks as variables before setting dependencies.

Use >> to say one task runs before another.

Use << to say one task runs after another.

Chain tasks with these operators to create clear workflows.