Trigger rules (all_success, one_success, none_failed) in Apache Airflow - Deep Dive

Overview - Trigger rules (all_success, one_success, none_failed)

What is it?

Trigger rules in Airflow decide when a task should run based on the status of its upstream tasks. They control task execution flow by checking if previous tasks succeeded, failed, or met other conditions. Common trigger rules include all_success, one_success, and none_failed. These rules help manage complex workflows by defining clear dependencies.

Why it matters

Without trigger rules, tasks would run blindly without considering the success or failure of previous tasks, leading to wasted resources or incorrect results. Trigger rules ensure workflows behave predictably and efficiently, preventing errors from cascading and enabling conditional task execution. This control is crucial for reliable automation in data pipelines and other processes.

Where it fits

Learners should first understand basic Airflow concepts like DAGs, tasks, and task dependencies. After mastering trigger rules, they can explore advanced workflow patterns, error handling, and dynamic task generation. This topic fits in the middle of the Airflow learning path, bridging basic task orchestration and complex workflow control.

Mental Model

Core Idea

Trigger rules are the conditions that decide if a task runs based on the success or failure of its upstream tasks.

Think of it like...

It's like deciding whether to start cooking a meal only if all ingredients are ready, or if at least one ingredient is prepared, or if none of the ingredients are spoiled.

┌───────────────┐
│ Upstream Tasks │
└──────┬────────┘
       │
       ▼
┌─────────────────────┐
│ Trigger Rule Checks  │
│ (all_success, etc.)  │
└─────────┬───────────┘
          │
          ▼
   ┌─────────────┐
   │ Run Task?   │
   └─────────────┘

Build-Up - 6 Steps

1

Foundation

Understanding Airflow Task Dependencies

Concept: Tasks in Airflow depend on others to define execution order.

In Airflow, tasks are connected in a Directed Acyclic Graph (DAG). Each task can have upstream tasks that must finish before it runs. By default, a task runs only if all its upstream tasks succeed.

Result

Tasks run in the order defined by dependencies, ensuring correct workflow sequence.

Understanding dependencies is essential because trigger rules build on how tasks relate to each other.

2

Foundation

Default Trigger Rule: all_success

Concept: The default trigger rule requires all upstream tasks to succeed before running.

By default, Airflow uses the 'all_success' trigger rule. This means a task runs only if every upstream task finished successfully. If any upstream task fails or is skipped, the task will not run.

Result

Tasks run only when all dependencies succeed, preventing errors from propagating.

Knowing the default behavior helps you understand why some tasks don't run when upstream tasks fail.

3

Intermediate

Trigger Rule: one_success Explained

🤔Before reading on: do you think 'one_success' runs a task if exactly one upstream task succeeds or if at least one succeeds? Commit to your answer.

Concept: 'one_success' runs a task if at least one upstream task succeeds, ignoring others' failures.

The 'one_success' trigger rule allows a task to run if any one of its upstream tasks succeeds. It ignores failures or skipped states of other upstream tasks. This is useful when multiple upstream tasks provide alternative paths.

Result

Tasks run even if some upstream tasks fail, as long as one succeeds.

Understanding 'one_success' enables flexible workflows where multiple paths can trigger the same task.

4

Intermediate

Trigger Rule: none_failed Explained

🤔Before reading on: does 'none_failed' allow a task to run if some upstream tasks are skipped? Commit to your answer.

Concept: 'none_failed' runs a task if no upstream tasks failed, but allows skipped tasks.

The 'none_failed' trigger rule lets a task run if none of its upstream tasks failed. It ignores skipped tasks, so the task can run even if some upstream tasks were skipped. This is useful when skipped tasks are acceptable but failures are not.

Result

Tasks run as long as no upstream task failed, increasing workflow resilience.

Knowing 'none_failed' helps handle workflows with optional or conditional tasks gracefully.

5

Advanced

Combining Trigger Rules for Complex Logic

🤔Before reading on: can you combine multiple trigger rules on a single task in Airflow? Commit to your answer.

Concept: Airflow allows setting one trigger rule per task, but complex logic can be built using task groups and branching.

While each task has a single trigger rule, you can design DAGs with branching operators and task groups to simulate complex conditions. For example, use BranchPythonOperator to choose paths, then tasks with different trigger rules to handle outcomes.

Result

You can create sophisticated workflows that react dynamically to task outcomes.

Understanding how to combine trigger rules with branching unlocks powerful workflow designs.

6

Expert

Trigger Rules Impact on Task Retries and SLA

🤔Before reading on: do trigger rules affect how retries and SLAs behave in Airflow? Commit to your answer.

Concept: Trigger rules influence task retries and SLA misses by controlling when tasks run after upstream failures or skips.

If a task's trigger rule allows running despite upstream failures, retries may occur even if upstream tasks failed. SLA misses can also be affected if tasks run unexpectedly or not at all. Understanding this helps in setting realistic SLAs and retry policies.

Result

Better control over task execution timing and alerting in production workflows.

Knowing trigger rules' effect on retries and SLAs prevents unexpected workflow behavior and alert fatigue.

Under the Hood

Airflow tracks the state of each task instance in its metadata database. When deciding to run a task, Airflow evaluates the trigger rule by checking the states of all upstream tasks. It uses this evaluation to determine if the task should be scheduled or skipped. This evaluation happens at runtime before task execution.

Why designed this way?

Trigger rules were designed to provide flexible control over task execution without complex custom code. They allow workflows to adapt to different success and failure scenarios, improving reliability and efficiency. Alternatives like manual state checks would be error-prone and less maintainable.

┌───────────────┐
│ Upstream Task │
│ States Stored │
│ in Metadata   │
└──────┬────────┘
       │
       ▼
┌───────────────────────────┐
│ Airflow Scheduler Queries  │
│ Upstream States & Applies  │
│ Trigger Rule Logic         │
└─────────────┬─────────────┘
              │
              ▼
       ┌─────────────┐
       │ Run or Skip │
       │ Task        │
       └─────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does 'all_success' mean a task runs if one upstream task is skipped? Commit yes or no.

Common Belief: If one upstream task is skipped, 'all_success' still allows the task to run.

Reality: 'all_success' requires all upstream tasks to succeed; skipped tasks cause the task not to run.

Why it matters: Misunderstanding this causes tasks to be skipped unexpectedly, breaking workflows.

Quick: Does 'one_success' run a task only if exactly one upstream task succeeds? Commit yes or no.

Common Belief: 'one_success' runs a task only if exactly one upstream task succeeds.

Reality: 'one_success' runs the task if at least one upstream task succeeds, regardless of others.

Why it matters: This misconception limits workflow flexibility and can cause confusion in task execution.

Quick: Does 'none_failed' allow a task to run if some upstream tasks failed? Commit yes or no.

Common Belief: 'none_failed' runs a task even if some upstream tasks failed, as long as others succeeded.

Reality: 'none_failed' runs a task only if no upstream tasks failed; any failure blocks the task.

Why it matters: Ignoring this leads to tasks running when they shouldn't, risking data corruption or errors.

Quick: Can a task have multiple trigger rules at once? Commit yes or no.

Common Belief: A task can have multiple trigger rules combined for complex conditions.

Reality: Each task can have only one trigger rule; complex logic requires DAG design patterns.

Why it matters: Expecting multiple trigger rules per task leads to design errors and unexpected behavior.

Expert Zone

1

Tasks with 'none_failed' can run even if some upstream tasks are skipped, which is useful for optional dependencies but can cause silent skips if not monitored.

2

Using 'one_success' in large DAGs can mask upstream failures, so it's important to combine it with alerting to catch issues early.

3

Trigger rules interact subtly with task retries and SLA misses, requiring careful configuration to avoid false alerts or missed runs.

When NOT to use

Avoid using 'one_success' when all upstream tasks must succeed for correctness; use 'all_success' instead. For workflows requiring complex conditional logic, use branching operators rather than relying solely on trigger rules. If you need to handle failure paths explicitly, consider using 'all_done' combined with sensors or custom operators.

Production Patterns

In production, teams use 'all_success' for critical sequential tasks, 'none_failed' for tasks that can tolerate skipped upstream tasks, and 'one_success' for fallback or alternative path tasks. Branching operators combined with trigger rules enable dynamic workflows that adapt to data availability or external events. Monitoring task states alongside trigger rules helps maintain reliability.

Connections

Conditional Statements in Programming

Trigger rules act like conditional checks deciding if a task runs, similar to if-else statements controlling code flow.

Understanding trigger rules as conditional logic helps grasp how workflows adapt dynamically based on task outcomes.

Project Management Dependencies

Trigger rules mirror task dependencies in project plans where some tasks start only after others succeed or fail.

Seeing trigger rules as project dependencies clarifies their role in orchestrating complex workflows reliably.

Biological Signal Pathways

Trigger rules resemble biological signals that activate processes only when certain conditions in the cell are met.

Recognizing this connection shows how conditional activation is a universal pattern in both technology and nature.

Common Pitfalls

#1 Task does not run because upstream task was skipped but trigger rule was 'all_success'.

Wrong approach:

task = PythonOperator(task_id='task', trigger_rule='all_success', ...)

Correct approach:

task = PythonOperator(task_id='task', trigger_rule='none_failed', ...)

Root cause:

Misunderstanding that 'all_success' requires all upstream tasks to succeed, not just not fail.

#2 Task runs unexpectedly despite upstream failures using 'one_success'.

Wrong approach:

task = PythonOperator(task_id='task', trigger_rule='one_success', ...)

Correct approach:

task = PythonOperator(task_id='task', trigger_rule='all_success', ...)

Root cause:

Assuming 'one_success' means all upstream tasks must succeed, not just one.

#3 Trying to assign multiple trigger rules to a single task.

Wrong approach:

task = PythonOperator(task_id='task', trigger_rule=['all_success', 'none_failed'], ...)

Correct approach:

Use branching operators or task groups to handle complex conditions instead of multiple trigger rules.

Root cause:

Believing Airflow supports multiple trigger rules per task, which it does not.

Key Takeaways

Trigger rules control when Airflow tasks run based on upstream task states, enabling flexible workflow management.

'all_success' is the default rule requiring all upstream tasks to succeed before running a task.

'one_success' allows a task to run if any upstream task succeeds, useful for alternative paths.

'none_failed' runs a task if no upstream tasks failed, tolerating skipped tasks for resilience.

Understanding trigger rules deeply helps design reliable, efficient, and maintainable workflows in Airflow.

Practice

1. In Airflow, which trigger rule ensures a task runs only if all upstream tasks have succeeded?

easy

A. all_success

B. one_success

C. none_failed

D. all_failed

Solution

1. Step 1: Understand the meaning of all_success

   This trigger rule requires every upstream task to complete successfully before the downstream task runs.

2. Step 2: Compare with other trigger rules

   one_success runs if any upstream succeeds, none_failed runs if no upstream failed (some may be skipped), and all_failed runs only if all upstream failed.

3. Final Answer:

   all_success -> Option A

4. Quick Check:

   All upstream success = all_success [OK]

Hint: All upstream must succeed means all_success [OK]

Common Mistakes:

• Confusing one_success with all_success
• Thinking none_failed means all succeeded
• Assuming all_failed runs on success

2. Which of the following is the correct way to set a task's trigger rule to run if at least one upstream task succeeds in Airflow DAG code?

easy

A. trigger_rule='all_success'

B. trigger_rule='all_done'

C. trigger_rule='none_failed'

D. trigger_rule='one_success'

Solution

1. Step 1: Recall the trigger rule for running on any upstream success

   The trigger rule one_success runs the task if at least one upstream task succeeded.

2. Step 2: Check syntax correctness

   Setting trigger_rule='one_success' is the correct syntax in Airflow task definition.

3. Final Answer:

   trigger_rule='one_success' -> Option D

4. Quick Check:

   At least one success = trigger_rule='one_success' [OK]

Hint: Use trigger_rule='one_success' for any upstream success [OK]

Common Mistakes:

• Using all_success instead of one_success
• Misspelling trigger_rule or using wrong quotes
• Confusing none_failed with one_success

3. Given a DAG with three upstream tasks: task1 (success), task2 (skipped), task3 (failed). If a downstream task uses trigger_rule='none_failed', will it run?

medium

A. Yes, because no upstream task failed

B. No, because one upstream task failed

C. Yes, because at least one upstream succeeded

D. No, because all upstream tasks must succeed

Solution

1. Step 1: Understand none_failed trigger rule

   This rule runs the task only if no upstream task has failed. If any upstream failed, it will not run.

2. Step 2: Analyze upstream task states

   task3 failed, so the condition of no failures is false. Therefore, the downstream task will not run.

3. Final Answer:

   No, because one upstream task failed -> Option B

4. Quick Check:

   Any failure blocks none_failed run [OK]

Hint: If any upstream failed, none_failed won't run [OK]

Common Mistakes:

• Ignoring failed upstream tasks when using none_failed
• Confusing skipped with failed states
• Assuming none_failed runs if some succeed

4. You have a task with trigger_rule='one_success' but it never runs even though some upstream tasks succeed. What is the likely cause?

medium

A. The task has no upstream dependencies defined

B. The DAG is paused

C. The trigger_rule is misspelled or not set correctly

D. All upstream tasks failed

Solution

1. Step 1: Check trigger_rule spelling and assignment

   If the trigger_rule is misspelled or not properly set, Airflow defaults to all_success, causing the task not to run if not all upstream succeed.

2. Step 2: Verify other conditions

   Since some upstream succeed, and the task has dependencies, the most common cause is incorrect trigger_rule syntax.

3. Final Answer:

   The trigger_rule is misspelled or not set correctly -> Option C

4. Quick Check:

   Incorrect trigger_rule causes unexpected task skipping [OK]

Hint: Check trigger_rule spelling if task doesn't run as expected [OK]

Common Mistakes:

• Assuming upstream failure when some succeeded
• Ignoring DAG paused state
• Not defining upstream dependencies

5. You want a downstream task to run if either all upstream tasks succeed or if none failed but some were skipped. Which trigger rule should you use?

hard

A. none_failed

B. one_success

C. all_success

D. all_done

Solution

1. Step 1: Understand the requirement

   The task should run if all upstream succeeded or if none failed but some were skipped. This means failures block the run, but skips do not.

2. Step 2: Match with trigger rules

   all_success requires all succeed (no skips allowed). none_failed allows skips but no failures. one_success runs if any succeed, which is too loose. all_done runs regardless of success or failure.

3. Step 3: Choose the correct trigger rule

   none_failed fits the requirement exactly.

4. Final Answer:

   none_failed -> Option A

5. Quick Check:

   No failures but allows skips = none_failed [OK]

Hint: Use none_failed to allow skips but no failures [OK]

Common Mistakes:

• Choosing all_success which disallows skips
• Using one_success which is too permissive
• Confusing all_done with failure conditions