Trigger rules (all_success, one_success, none_failed) in Apache Airflow - Commands & Configuration

Introduction

In Airflow, tasks can depend on each other. Trigger rules decide when a task should run based on the success or failure of previous tasks. This helps control the flow of work in a clear way.

When you want a task to run only if all previous tasks succeeded.

When you want a task to run if at least one previous task succeeded.

When you want a task to run as long as no previous tasks failed.

When you want to handle different outcomes of parallel tasks.

When you want to avoid running tasks if any upstream task failed.

Config File - example_dag.py

example_dag.py

from airflow import DAG
from airflow.operators.bash import BashOperator
from airflow.utils.dates import days_ago
from airflow.utils.trigger_rule import TriggerRule

default_args = {
    'start_date': days_ago(1),
}

dag = DAG(
    'trigger_rule_example',
    default_args=default_args,
    schedule_interval=None,
    catchup=False
)

# Task that always succeeds
success_task = BashOperator(
    task_id='success_task',
    bash_command='echo Success',
    dag=dag
)

# Task that always fails
fail_task = BashOperator(
    task_id='fail_task',
    bash_command='exit 1',
    dag=dag
)

# Task runs only if all upstream tasks succeed
all_success_task = BashOperator(
    task_id='all_success_task',
    bash_command='echo All upstream succeeded',
    trigger_rule=TriggerRule.ALL_SUCCESS,
    dag=dag
)

# Task runs if at least one upstream task succeeds
one_success_task = BashOperator(
    task_id='one_success_task',
    bash_command='echo At least one upstream succeeded',
    trigger_rule=TriggerRule.ONE_SUCCESS,
    dag=dag
)

# Task runs if no upstream tasks failed
none_failed_task = BashOperator(
    task_id='none_failed_task',
    bash_command='echo No upstream failed',
    trigger_rule=TriggerRule.NONE_FAILED,
    dag=dag
)

success_task >> [all_success_task, one_success_task, none_failed_task]
fail_task >> [all_success_task, one_success_task, none_failed_task]

This DAG defines five tasks:

• success_task: always succeeds.
• fail_task: always fails.
• all_success_task: runs only if all upstream tasks succeed (all_success).
• one_success_task: runs if at least one upstream task succeeds (one_success).
• none_failed_task: runs if no upstream tasks failed (none_failed).

The last three tasks depend on both success_task and fail_task to show how trigger rules affect execution.

Commands

List all available DAGs to confirm the DAG is recognized by Airflow.

Terminal

airflow dags list

Expected Output

trigger_rule_example

Trigger the DAG run manually to start the tasks and observe trigger rules in action.

Terminal

airflow dags trigger trigger_rule_example

Expected Output

Created <DagRun trigger_rule_example @ 2024-06-01T00:00:00+00:00: manual__2024-06-01T00:00:00+00:00, externally triggered: True>

List all tasks in the DAG to verify task names and dependencies.

Terminal

airflow tasks list trigger_rule_example

Expected Output

success_task

fail_task

all_success_task

one_success_task

none_failed_task

Check the state of the success_task after DAG run to confirm it succeeded.

Terminal

airflow tasks state trigger_rule_example success_task manual__2024-06-01T00:00:00+00:00

Expected Output

success

Check the state of the fail_task after DAG run to confirm it failed.

Terminal

airflow tasks state trigger_rule_example fail_task manual__2024-06-01T00:00:00+00:00

Expected Output

failed

Check the state of all_success_task to see if it ran (it should not because one upstream failed).

Terminal

airflow tasks state trigger_rule_example all_success_task manual__2024-06-01T00:00:00+00:00

Expected Output

skipped

Check the state of one_success_task to see if it ran (it should run because one upstream succeeded).

Terminal

airflow tasks state trigger_rule_example one_success_task manual__2024-06-01T00:00:00+00:00

Expected Output

success

Check the state of none_failed_task to see if it ran (it should not because one upstream failed).

Terminal

airflow tasks state trigger_rule_example none_failed_task manual__2024-06-01T00:00:00+00:00

Expected Output

skipped

Key Concept

Trigger rules control when a task runs based on the success or failure of its upstream tasks.

Common Mistakes

Setting trigger_rule to ALL_SUCCESS but expecting the task to run when any upstream succeeds.

ALL_SUCCESS requires every upstream task to succeed, so the task will be skipped if any fail.

Use ONE_SUCCESS if you want the task to run when at least one upstream task succeeds.

Not setting trigger_rule explicitly and assuming default behavior covers all cases.

The default trigger_rule is ALL_SUCCESS, which may skip tasks if any upstream fails.

Set trigger_rule explicitly to NONE_FAILED or ONE_SUCCESS when needed.

Confusing NONE_FAILED with ALL_SUCCESS and expecting the same behavior.

NONE_FAILED runs if no upstream failed, even if some upstream tasks were skipped, unlike ALL_SUCCESS.

Choose NONE_FAILED when you want to run tasks despite skipped upstream tasks but no failures.

Summary

Define trigger rules in tasks to control execution based on upstream task results.

ALL_SUCCESS runs a task only if all upstream tasks succeed.

ONE_SUCCESS runs a task if at least one upstream task succeeds.

NONE_FAILED runs a task if no upstream tasks failed, even if some were skipped.