Trigger rules (all_success, one_success, none_failed) in Apache Airflow - Time & Space Complexity

Time Complexity: Trigger rules (all_success, one_success, none_failed)

O(n)

Understanding Time Complexity

When using trigger rules in Airflow, it's important to understand how the number of tasks affects the time it takes to decide if a task should run.

We want to know how the checking time grows as more tasks complete before the next task runs.

Scenario Under Consideration

Analyze the time complexity of this Airflow task trigger rule check.

from airflow import DAG
from airflow.operators.dummy import DummyOperator
from airflow.utils.trigger_rule import TriggerRule
from datetime import datetime

dag = DAG('example_trigger_rule', start_date=datetime(2024, 1, 1))

start = DummyOperator(task_id='start', dag=dag)

check = DummyOperator(
    task_id='check',
    dag=dag,
    trigger_rule=TriggerRule.ALL_SUCCESS
)

start >> check

This code sets a task with a trigger rule that waits for all upstream tasks to succeed before running.

Identify Repeating Operations

When Airflow decides if the task can run, it checks the status of all upstream tasks.

• Primary operation: Checking each upstream task's state.
• How many times: Once per upstream task, so as many times as there are upstream tasks.

How Execution Grows With Input

As the number of upstream tasks increases, the time to check their states grows proportionally.

Input Size (n)	Approx. Operations
10	10 checks
100	100 checks
1000	1000 checks

Pattern observation: The checking time grows linearly with the number of upstream tasks.

Final Time Complexity

Time Complexity: O(n)

This means the time to decide if a task runs grows directly with the number of upstream tasks it depends on.

Common Mistake

[X] Wrong: "The trigger rule check happens instantly no matter how many tasks there are."

[OK] Correct: Each upstream task's state must be checked, so more tasks mean more checks and more time.

Interview Connect

Understanding how trigger rules scale helps you design workflows that run efficiently and predictably as they grow.

Self-Check

What if the trigger rule was changed to one_success instead of all_success? How would the time complexity change?

Practice

1. In Airflow, which trigger rule ensures a task runs only if all upstream tasks have succeeded?

easy

A. all_success

B. one_success

C. none_failed

D. all_failed

Solution

1. Step 1: Understand the meaning of all_success

   This trigger rule requires every upstream task to complete successfully before the downstream task runs.

2. Step 2: Compare with other trigger rules

   one_success runs if any upstream succeeds, none_failed runs if no upstream failed (some may be skipped), and all_failed runs only if all upstream failed.

3. Final Answer:

   all_success -> Option A

4. Quick Check:

   All upstream success = all_success [OK]

Hint: All upstream must succeed means all_success [OK]

Common Mistakes:

• Confusing one_success with all_success
• Thinking none_failed means all succeeded
• Assuming all_failed runs on success

2. Which of the following is the correct way to set a task's trigger rule to run if at least one upstream task succeeds in Airflow DAG code?

easy

A. trigger_rule='all_success'

B. trigger_rule='all_done'

C. trigger_rule='none_failed'

D. trigger_rule='one_success'

Solution

1. Step 1: Recall the trigger rule for running on any upstream success

   The trigger rule one_success runs the task if at least one upstream task succeeded.

2. Step 2: Check syntax correctness

   Setting trigger_rule='one_success' is the correct syntax in Airflow task definition.

3. Final Answer:

   trigger_rule='one_success' -> Option D

4. Quick Check:

   At least one success = trigger_rule='one_success' [OK]

Hint: Use trigger_rule='one_success' for any upstream success [OK]

Common Mistakes:

• Using all_success instead of one_success
• Misspelling trigger_rule or using wrong quotes
• Confusing none_failed with one_success

3. Given a DAG with three upstream tasks: task1 (success), task2 (skipped), task3 (failed). If a downstream task uses trigger_rule='none_failed', will it run?

medium

A. Yes, because no upstream task failed

B. No, because one upstream task failed

C. Yes, because at least one upstream succeeded

D. No, because all upstream tasks must succeed

Solution

1. Step 1: Understand none_failed trigger rule

   This rule runs the task only if no upstream task has failed. If any upstream failed, it will not run.

2. Step 2: Analyze upstream task states

   task3 failed, so the condition of no failures is false. Therefore, the downstream task will not run.

3. Final Answer:

   No, because one upstream task failed -> Option B

4. Quick Check:

   Any failure blocks none_failed run [OK]

Hint: If any upstream failed, none_failed won't run [OK]

Common Mistakes:

• Ignoring failed upstream tasks when using none_failed
• Confusing skipped with failed states
• Assuming none_failed runs if some succeed

4. You have a task with trigger_rule='one_success' but it never runs even though some upstream tasks succeed. What is the likely cause?

medium

A. The task has no upstream dependencies defined

B. The DAG is paused

C. The trigger_rule is misspelled or not set correctly

D. All upstream tasks failed

Solution

1. Step 1: Check trigger_rule spelling and assignment

   If the trigger_rule is misspelled or not properly set, Airflow defaults to all_success, causing the task not to run if not all upstream succeed.

2. Step 2: Verify other conditions

   Since some upstream succeed, and the task has dependencies, the most common cause is incorrect trigger_rule syntax.

3. Final Answer:

   The trigger_rule is misspelled or not set correctly -> Option C

4. Quick Check:

   Incorrect trigger_rule causes unexpected task skipping [OK]

Hint: Check trigger_rule spelling if task doesn't run as expected [OK]

Common Mistakes:

• Assuming upstream failure when some succeeded
• Ignoring DAG paused state
• Not defining upstream dependencies

5. You want a downstream task to run if either all upstream tasks succeed or if none failed but some were skipped. Which trigger rule should you use?

hard

A. none_failed

B. one_success

C. all_success

D. all_done

Solution

1. Step 1: Understand the requirement

   The task should run if all upstream succeeded or if none failed but some were skipped. This means failures block the run, but skips do not.

2. Step 2: Match with trigger rules

   all_success requires all succeed (no skips allowed). none_failed allows skips but no failures. one_success runs if any succeed, which is too loose. all_done runs regardless of success or failure.

3. Step 3: Choose the correct trigger rule

   none_failed fits the requirement exactly.

4. Final Answer:

   none_failed -> Option A

5. Quick Check:

   No failures but allows skips = none_failed [OK]

Hint: Use none_failed to allow skips but no failures [OK]

Common Mistakes:

• Choosing all_success which disallows skips
• Using one_success which is too permissive
• Confusing all_done with failure conditions