Trigger rules (all_success, one_success, none_failed) in Apache Airflow - Mini Project: Build & Apply

Airflow Trigger Rules Basics

📖 Scenario: You are managing a data pipeline in Airflow. You want to control when tasks run based on the success or failure of other tasks.

🎯 Goal: Build a simple Airflow DAG with five tasks and apply different trigger_rule settings: all_success, one_success, and none_failed.

📋 What You'll Learn

Create a DAG named trigger_rule_demo

Add three dummy tasks: task_a, task_b, and task_c

Set task_c to run only if task_a and task_b both succeed (all_success)

Add a fourth dummy task task_d that runs if at least one of task_a or task_b succeeds (one_success)

Add a fifth dummy task task_e that runs if none of task_a or task_b failed (none_failed)

💡 Why This Matters

🌍 Real World

In real data pipelines, you often want tasks to run only if certain previous tasks succeeded or did not fail. Trigger rules help control this flow.

💼 Career

Understanding trigger rules is essential for Airflow users to build reliable and efficient workflows in data engineering and automation jobs.

1

Create the DAG and initial tasks

Create an Airflow DAG named trigger_rule_demo with default arguments. Add two dummy tasks called task_a and task_b using DummyOperator.

Apache Airflow

from airflow import DAG
from airflow.operators.dummy import DummyOperator
from datetime import datetime

# Your code here

Hint

Use DAG to create the DAG and DummyOperator for tasks.

2

Add task_c with trigger_rule all_success

Add a dummy task called task_c that depends on task_a and task_b. Set its trigger_rule to all_success so it runs only if both task_a and task_b succeed.

Apache Airflow

from airflow import DAG
from airflow.operators.dummy import DummyOperator
from datetime import datetime

default_args = {
    'start_date': datetime(2024, 1, 1)
}

dag = DAG('trigger_rule_demo', default_args=default_args, schedule_interval='@daily')

task_a = DummyOperator(task_id='task_a', dag=dag)
task_b = DummyOperator(task_id='task_b', dag=dag)

# Your code here

Hint

Use trigger_rule='all_success' in DummyOperator and set dependencies with >>.

3

Add task_d with trigger_rule one_success

Add a dummy task called task_d that depends on task_a and task_b. Set its trigger_rule to one_success so it runs if at least one of task_a or task_b succeeds.

Apache Airflow

from airflow import DAG
from airflow.operators.dummy import DummyOperator
from datetime import datetime

default_args = {
    'start_date': datetime(2024, 1, 1)
}

dag = DAG('trigger_rule_demo', default_args=default_args, schedule_interval='@daily')

task_a = DummyOperator(task_id='task_a', dag=dag)
task_b = DummyOperator(task_id='task_b', dag=dag)

task_c = DummyOperator(task_id='task_c', trigger_rule='all_success', dag=dag)

task_a >> task_c
task_b >> task_c

# Your code here

Hint

Use trigger_rule='one_success' and set dependencies similarly to task_c.

4

Add task_e with trigger_rule none_failed

Add a dummy task called task_e that depends on task_a and task_b. Set its trigger_rule to none_failed so it runs if none of task_a or task_b failed.

Apache Airflow

from airflow import DAG
from airflow.operators.dummy import DummyOperator
from datetime import datetime

default_args = {
    'start_date': datetime(2024, 1, 1)
}

dag = DAG('trigger_rule_demo', default_args=default_args, schedule_interval='@daily')

task_a = DummyOperator(task_id='task_a', dag=dag)
task_b = DummyOperator(task_id='task_b', dag=dag)

task_c = DummyOperator(task_id='task_c', trigger_rule='all_success', dag=dag)

task_a >> task_c
task_b >> task_c

task_d = DummyOperator(task_id='task_d', trigger_rule='one_success', dag=dag)

task_a >> task_d
task_b >> task_d

# Your code here

Hint

Use trigger_rule='none_failed' and set dependencies like previous tasks.

Practice

1. In Airflow, which trigger rule ensures a task runs only if all upstream tasks have succeeded?

easy

A. all_success

B. one_success

C. none_failed

D. all_failed

Solution

1. Step 1: Understand the meaning of all_success

   This trigger rule requires every upstream task to complete successfully before the downstream task runs.

2. Step 2: Compare with other trigger rules

   one_success runs if any upstream succeeds, none_failed runs if no upstream failed (some may be skipped), and all_failed runs only if all upstream failed.

3. Final Answer:

   all_success -> Option A

4. Quick Check:

   All upstream success = all_success [OK]

Hint: All upstream must succeed means all_success [OK]

Common Mistakes:

• Confusing one_success with all_success
• Thinking none_failed means all succeeded
• Assuming all_failed runs on success

2. Which of the following is the correct way to set a task's trigger rule to run if at least one upstream task succeeds in Airflow DAG code?

easy

A. trigger_rule='all_success'

B. trigger_rule='all_done'

C. trigger_rule='none_failed'

D. trigger_rule='one_success'

Solution

1. Step 1: Recall the trigger rule for running on any upstream success

   The trigger rule one_success runs the task if at least one upstream task succeeded.

2. Step 2: Check syntax correctness

   Setting trigger_rule='one_success' is the correct syntax in Airflow task definition.

3. Final Answer:

   trigger_rule='one_success' -> Option D

4. Quick Check:

   At least one success = trigger_rule='one_success' [OK]

Hint: Use trigger_rule='one_success' for any upstream success [OK]

Common Mistakes:

• Using all_success instead of one_success
• Misspelling trigger_rule or using wrong quotes
• Confusing none_failed with one_success

3. Given a DAG with three upstream tasks: task1 (success), task2 (skipped), task3 (failed). If a downstream task uses trigger_rule='none_failed', will it run?

medium

A. Yes, because no upstream task failed

B. No, because one upstream task failed

C. Yes, because at least one upstream succeeded

D. No, because all upstream tasks must succeed

Solution

1. Step 1: Understand none_failed trigger rule

   This rule runs the task only if no upstream task has failed. If any upstream failed, it will not run.

2. Step 2: Analyze upstream task states

   task3 failed, so the condition of no failures is false. Therefore, the downstream task will not run.

3. Final Answer:

   No, because one upstream task failed -> Option B

4. Quick Check:

   Any failure blocks none_failed run [OK]

Hint: If any upstream failed, none_failed won't run [OK]

Common Mistakes:

• Ignoring failed upstream tasks when using none_failed
• Confusing skipped with failed states
• Assuming none_failed runs if some succeed

4. You have a task with trigger_rule='one_success' but it never runs even though some upstream tasks succeed. What is the likely cause?

medium

A. The task has no upstream dependencies defined

B. The DAG is paused

C. The trigger_rule is misspelled or not set correctly

D. All upstream tasks failed

Solution

1. Step 1: Check trigger_rule spelling and assignment

   If the trigger_rule is misspelled or not properly set, Airflow defaults to all_success, causing the task not to run if not all upstream succeed.

2. Step 2: Verify other conditions

   Since some upstream succeed, and the task has dependencies, the most common cause is incorrect trigger_rule syntax.

3. Final Answer:

   The trigger_rule is misspelled or not set correctly -> Option C

4. Quick Check:

   Incorrect trigger_rule causes unexpected task skipping [OK]

Hint: Check trigger_rule spelling if task doesn't run as expected [OK]

Common Mistakes:

• Assuming upstream failure when some succeeded
• Ignoring DAG paused state
• Not defining upstream dependencies

5. You want a downstream task to run if either all upstream tasks succeed or if none failed but some were skipped. Which trigger rule should you use?

hard

A. none_failed

B. one_success

C. all_success

D. all_done

Solution

1. Step 1: Understand the requirement

   The task should run if all upstream succeeded or if none failed but some were skipped. This means failures block the run, but skips do not.

2. Step 2: Match with trigger rules

   all_success requires all succeed (no skips allowed). none_failed allows skips but no failures. one_success runs if any succeed, which is too loose. all_done runs regardless of success or failure.

3. Step 3: Choose the correct trigger rule

   none_failed fits the requirement exactly.

4. Final Answer:

   none_failed -> Option A

5. Quick Check:

   No failures but allows skips = none_failed [OK]

Hint: Use none_failed to allow skips but no failures [OK]

Common Mistakes:

• Choosing all_success which disallows skips
• Using one_success which is too permissive
• Confusing all_done with failure conditions