Trigger rules (all_success, one_success, none_failed) in Apache Airflow - Deep Dive

Overview - Trigger rules (all_success, one_success, none_failed)

What is it?

Trigger rules in Airflow decide when a task should run based on the status of its upstream tasks. They control task execution flow by checking if previous tasks succeeded, failed, or met other conditions. Common trigger rules include all_success, one_success, and none_failed. These rules help manage complex workflows by defining clear dependencies.

Why it matters

Without trigger rules, tasks would run blindly without considering the success or failure of previous tasks, leading to wasted resources or incorrect results. Trigger rules ensure workflows behave predictably and efficiently, preventing errors from cascading and enabling conditional task execution. This control is crucial for reliable automation in data pipelines and other processes.

Where it fits

Learners should first understand basic Airflow concepts like DAGs, tasks, and task dependencies. After mastering trigger rules, they can explore advanced workflow patterns, error handling, and dynamic task generation. This topic fits in the middle of the Airflow learning path, bridging basic task orchestration and complex workflow control.

Mental Model

Core Idea

Trigger rules are the conditions that decide if a task runs based on the success or failure of its upstream tasks.

Think of it like...

It's like deciding whether to start cooking a meal only if all ingredients are ready, or if at least one ingredient is prepared, or if none of the ingredients are spoiled.

┌───────────────┐
│ Upstream Tasks │
└──────┬────────┘
       │
       ▼
┌─────────────────────┐
│ Trigger Rule Checks  │
│ (all_success, etc.)  │
└─────────┬───────────┘
          │
          ▼
   ┌─────────────┐
   │ Run Task?   │
   └─────────────┘

Build-Up - 6 Steps

1

Foundation

Understanding Airflow Task Dependencies

Concept: Tasks in Airflow depend on others to define execution order.

In Airflow, tasks are connected in a Directed Acyclic Graph (DAG). Each task can have upstream tasks that must finish before it runs. By default, a task runs only if all its upstream tasks succeed.

Result

Tasks run in the order defined by dependencies, ensuring correct workflow sequence.

Understanding dependencies is essential because trigger rules build on how tasks relate to each other.

2

Foundation

Default Trigger Rule: all_success

Concept: The default trigger rule requires all upstream tasks to succeed before running.

By default, Airflow uses the 'all_success' trigger rule. This means a task runs only if every upstream task finished successfully. If any upstream task fails or is skipped, the task will not run.

Result

Tasks run only when all dependencies succeed, preventing errors from propagating.

Knowing the default behavior helps you understand why some tasks don't run when upstream tasks fail.

3

Intermediate

Trigger Rule: one_success Explained

🤔Before reading on: do you think 'one_success' runs a task if exactly one upstream task succeeds or if at least one succeeds? Commit to your answer.

Concept: 'one_success' runs a task if at least one upstream task succeeds, ignoring others' failures.

The 'one_success' trigger rule allows a task to run if any one of its upstream tasks succeeds. It ignores failures or skipped states of other upstream tasks. This is useful when multiple upstream tasks provide alternative paths.

Result

Tasks run even if some upstream tasks fail, as long as one succeeds.

Understanding 'one_success' enables flexible workflows where multiple paths can trigger the same task.

4

Intermediate

Trigger Rule: none_failed Explained

🤔Before reading on: does 'none_failed' allow a task to run if some upstream tasks are skipped? Commit to your answer.

Concept: 'none_failed' runs a task if no upstream tasks failed, but allows skipped tasks.

The 'none_failed' trigger rule lets a task run if none of its upstream tasks failed. It ignores skipped tasks, so the task can run even if some upstream tasks were skipped. This is useful when skipped tasks are acceptable but failures are not.

Result

Tasks run as long as no upstream task failed, increasing workflow resilience.

Knowing 'none_failed' helps handle workflows with optional or conditional tasks gracefully.

5

Advanced

Combining Trigger Rules for Complex Logic

🤔Before reading on: can you combine multiple trigger rules on a single task in Airflow? Commit to your answer.

Concept: Airflow allows setting one trigger rule per task, but complex logic can be built using task groups and branching.

While each task has a single trigger rule, you can design DAGs with branching operators and task groups to simulate complex conditions. For example, use BranchPythonOperator to choose paths, then tasks with different trigger rules to handle outcomes.

Result

You can create sophisticated workflows that react dynamically to task outcomes.

Understanding how to combine trigger rules with branching unlocks powerful workflow designs.

6

Expert

Trigger Rules Impact on Task Retries and SLA

🤔Before reading on: do trigger rules affect how retries and SLAs behave in Airflow? Commit to your answer.

Concept: Trigger rules influence task retries and SLA misses by controlling when tasks run after upstream failures or skips.

If a task's trigger rule allows running despite upstream failures, retries may occur even if upstream tasks failed. SLA misses can also be affected if tasks run unexpectedly or not at all. Understanding this helps in setting realistic SLAs and retry policies.

Result

Better control over task execution timing and alerting in production workflows.

Knowing trigger rules' effect on retries and SLAs prevents unexpected workflow behavior and alert fatigue.

Under the Hood

Airflow tracks the state of each task instance in its metadata database. When deciding to run a task, Airflow evaluates the trigger rule by checking the states of all upstream tasks. It uses this evaluation to determine if the task should be scheduled or skipped. This evaluation happens at runtime before task execution.

Why designed this way?

Trigger rules were designed to provide flexible control over task execution without complex custom code. They allow workflows to adapt to different success and failure scenarios, improving reliability and efficiency. Alternatives like manual state checks would be error-prone and less maintainable.

┌───────────────┐
│ Upstream Task │
│ States Stored │
│ in Metadata   │
└──────┬────────┘
       │
       ▼
┌───────────────────────────┐
│ Airflow Scheduler Queries  │
│ Upstream States & Applies  │
│ Trigger Rule Logic         │
└─────────────┬─────────────┘
              │
              ▼
       ┌─────────────┐
       │ Run or Skip │
       │ Task        │
       └─────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does 'all_success' mean a task runs if one upstream task is skipped? Commit yes or no.

Common Belief: If one upstream task is skipped, 'all_success' still allows the task to run.

Reality: 'all_success' requires all upstream tasks to succeed; skipped tasks cause the task not to run.

Why it matters: Misunderstanding this causes tasks to be skipped unexpectedly, breaking workflows.

Quick: Does 'one_success' run a task only if exactly one upstream task succeeds? Commit yes or no.

Common Belief: 'one_success' runs a task only if exactly one upstream task succeeds.

Reality: 'one_success' runs the task if at least one upstream task succeeds, regardless of others.

Why it matters: This misconception limits workflow flexibility and can cause confusion in task execution.

Quick: Does 'none_failed' allow a task to run if some upstream tasks failed? Commit yes or no.

Common Belief: 'none_failed' runs a task even if some upstream tasks failed, as long as others succeeded.

Reality: 'none_failed' runs a task only if no upstream tasks failed; any failure blocks the task.

Why it matters: Ignoring this leads to tasks running when they shouldn't, risking data corruption or errors.

Quick: Can a task have multiple trigger rules at once? Commit yes or no.

Common Belief: A task can have multiple trigger rules combined for complex conditions.

Reality: Each task can have only one trigger rule; complex logic requires DAG design patterns.

Why it matters: Expecting multiple trigger rules per task leads to design errors and unexpected behavior.

Expert Zone

1

Tasks with 'none_failed' can run even if some upstream tasks are skipped, which is useful for optional dependencies but can cause silent skips if not monitored.

2

Using 'one_success' in large DAGs can mask upstream failures, so it's important to combine it with alerting to catch issues early.

3

Trigger rules interact subtly with task retries and SLA misses, requiring careful configuration to avoid false alerts or missed runs.

When NOT to use

Avoid using 'one_success' when all upstream tasks must succeed for correctness; use 'all_success' instead. For workflows requiring complex conditional logic, use branching operators rather than relying solely on trigger rules. If you need to handle failure paths explicitly, consider using 'all_done' combined with sensors or custom operators.

Production Patterns

In production, teams use 'all_success' for critical sequential tasks, 'none_failed' for tasks that can tolerate skipped upstream tasks, and 'one_success' for fallback or alternative path tasks. Branching operators combined with trigger rules enable dynamic workflows that adapt to data availability or external events. Monitoring task states alongside trigger rules helps maintain reliability.

Connections

Conditional Statements in Programming

Trigger rules act like conditional checks deciding if a task runs, similar to if-else statements controlling code flow.

Understanding trigger rules as conditional logic helps grasp how workflows adapt dynamically based on task outcomes.

Project Management Dependencies

Trigger rules mirror task dependencies in project plans where some tasks start only after others succeed or fail.

Seeing trigger rules as project dependencies clarifies their role in orchestrating complex workflows reliably.

Biological Signal Pathways

Trigger rules resemble biological signals that activate processes only when certain conditions in the cell are met.

Recognizing this connection shows how conditional activation is a universal pattern in both technology and nature.

Common Pitfalls

#1 Task does not run because upstream task was skipped but trigger rule was 'all_success'.

Wrong approach:

task = PythonOperator(task_id='task', trigger_rule='all_success', ...)

Correct approach:

task = PythonOperator(task_id='task', trigger_rule='none_failed', ...)

Root cause:

Misunderstanding that 'all_success' requires all upstream tasks to succeed, not just not fail.

#2 Task runs unexpectedly despite upstream failures using 'one_success'.

Wrong approach:

task = PythonOperator(task_id='task', trigger_rule='one_success', ...)

Correct approach:

task = PythonOperator(task_id='task', trigger_rule='all_success', ...)

Root cause:

Assuming 'one_success' means all upstream tasks must succeed, not just one.

#3 Trying to assign multiple trigger rules to a single task.

Wrong approach:

task = PythonOperator(task_id='task', trigger_rule=['all_success', 'none_failed'], ...)

Correct approach:

Use branching operators or task groups to handle complex conditions instead of multiple trigger rules.

Root cause:

Believing Airflow supports multiple trigger rules per task, which it does not.

Key Takeaways

Trigger rules control when Airflow tasks run based on upstream task states, enabling flexible workflow management.

'all_success' is the default rule requiring all upstream tasks to succeed before running a task.

'one_success' allows a task to run if any upstream task succeeds, useful for alternative paths.

'none_failed' runs a task if no upstream tasks failed, tolerating skipped tasks for resilience.

Understanding trigger rules deeply helps design reliable, efficient, and maintainable workflows in Airflow.