Trigger rules (all_success, one_success, none_failed) in Apache Airflow - Mini Project: Build & Apply

Airflow Trigger Rules Basics

📖 Scenario: You are managing a data pipeline in Airflow. You want to control when tasks run based on the success or failure of other tasks.

🎯 Goal: Build a simple Airflow DAG with five tasks and apply different trigger_rule settings: all_success, one_success, and none_failed.

📋 What You'll Learn

Create a DAG named trigger_rule_demo

Add three dummy tasks: task_a, task_b, and task_c

Set task_c to run only if task_a and task_b both succeed (all_success)

Add a fourth dummy task task_d that runs if at least one of task_a or task_b succeeds (one_success)

Add a fifth dummy task task_e that runs if none of task_a or task_b failed (none_failed)

💡 Why This Matters

🌍 Real World

In real data pipelines, you often want tasks to run only if certain previous tasks succeeded or did not fail. Trigger rules help control this flow.

💼 Career

Understanding trigger rules is essential for Airflow users to build reliable and efficient workflows in data engineering and automation jobs.

1

Create the DAG and initial tasks

Create an Airflow DAG named trigger_rule_demo with default arguments. Add two dummy tasks called task_a and task_b using DummyOperator.

Apache Airflow

from airflow import DAG
from airflow.operators.dummy import DummyOperator
from datetime import datetime

# Your code here

Need a hint?

Use DAG to create the DAG and DummyOperator for tasks.

2

Add task_c with trigger_rule all_success

Add a dummy task called task_c that depends on task_a and task_b. Set its trigger_rule to all_success so it runs only if both task_a and task_b succeed.

Apache Airflow

from airflow import DAG
from airflow.operators.dummy import DummyOperator
from datetime import datetime

default_args = {
    'start_date': datetime(2024, 1, 1)
}

dag = DAG('trigger_rule_demo', default_args=default_args, schedule_interval='@daily')

task_a = DummyOperator(task_id='task_a', dag=dag)
task_b = DummyOperator(task_id='task_b', dag=dag)

# Your code here

Need a hint?

Use trigger_rule='all_success' in DummyOperator and set dependencies with >>.

3

Add task_d with trigger_rule one_success

Add a dummy task called task_d that depends on task_a and task_b. Set its trigger_rule to one_success so it runs if at least one of task_a or task_b succeeds.

Apache Airflow

from airflow import DAG
from airflow.operators.dummy import DummyOperator
from datetime import datetime

default_args = {
    'start_date': datetime(2024, 1, 1)
}

dag = DAG('trigger_rule_demo', default_args=default_args, schedule_interval='@daily')

task_a = DummyOperator(task_id='task_a', dag=dag)
task_b = DummyOperator(task_id='task_b', dag=dag)

task_c = DummyOperator(task_id='task_c', trigger_rule='all_success', dag=dag)

task_a >> task_c
task_b >> task_c

# Your code here

Need a hint?

Use trigger_rule='one_success' and set dependencies similarly to task_c.

4

Add task_e with trigger_rule none_failed

Add a dummy task called task_e that depends on task_a and task_b. Set its trigger_rule to none_failed so it runs if none of task_a or task_b failed.

Apache Airflow

from airflow import DAG
from airflow.operators.dummy import DummyOperator
from datetime import datetime

default_args = {
    'start_date': datetime(2024, 1, 1)
}

dag = DAG('trigger_rule_demo', default_args=default_args, schedule_interval='@daily')

task_a = DummyOperator(task_id='task_a', dag=dag)
task_b = DummyOperator(task_id='task_b', dag=dag)

task_c = DummyOperator(task_id='task_c', trigger_rule='all_success', dag=dag)

task_a >> task_c
task_b >> task_c

task_d = DummyOperator(task_id='task_d', trigger_rule='one_success', dag=dag)

task_a >> task_d
task_b >> task_d

# Your code here

Need a hint?

Use trigger_rule='none_failed' and set dependencies like previous tasks.