Catchup and backfill behavior in Apache Airflow - Mini Project: Build & Apply

Understanding Catchup and Backfill Behavior in Airflow

📖 Scenario: You are managing scheduled data pipelines using Apache Airflow. Sometimes, your pipelines miss runs due to downtime or errors. You want to understand how Airflow handles these missed runs using catchup and backfill features.

🎯 Goal: Build a simple Airflow DAG that demonstrates how catchup controls whether missed DAG runs are executed automatically, and how to trigger backfill manually.

📋 What You'll Learn

Create an Airflow DAG with a fixed schedule

Set the catchup parameter to control automatic execution of missed runs

Use a BashOperator to simulate a task

Manually trigger backfill using Airflow CLI

Print logs to observe execution dates

💡 Why This Matters

🌍 Real World

In real data engineering projects, pipelines often miss scheduled runs due to system downtime or failures. Understanding catchup and backfill helps ensure data pipelines stay consistent and up to date.

💼 Career

DevOps and data engineers use Airflow catchup and backfill features to manage workflow reliability and data freshness, which are critical for business reporting and analytics.

1

Create a basic Airflow DAG with a BashOperator

Create a DAG named example_catchup_dag with a daily schedule starting from 2024-04-01. Inside the DAG, create a task called print_date using BashOperator that runs the command date.

Apache Airflow

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime

# Your code here

Need a hint?

Use with DAG(...) to define the DAG and include catchup=True to enable catchup.

2

Change the DAG to disable catchup

Modify the DAG to set catchup=False so that Airflow will not automatically run missed DAG runs.

Apache Airflow

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime

with DAG(
    dag_id='example_catchup_dag',
    start_date=datetime(2024, 4, 1),
    schedule_interval='@daily',
    catchup=False  # Change catchup here
) as dag:
    print_date = BashOperator(
        task_id='print_date',
        bash_command='date'
    )

Need a hint?

Set catchup=False inside the DAG arguments.

3

Add a task to print the execution date

Modify the print_date task's bash_command to print the execution date using the Airflow macro {{ ds }}. Use the command echo {{ ds }}.

Apache Airflow

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime

with DAG(
    dag_id='example_catchup_dag',
    start_date=datetime(2024, 4, 1),
    schedule_interval='@daily',
    catchup=False
) as dag:
    print_date = BashOperator(
        task_id='print_date',
        bash_command='echo {{ ds }}'  # Change command to print execution date
    )

Need a hint?

Use bash_command='echo {{ ds }}' to print the execution date.

4

Print a message explaining catchup and backfill

Write a print statement that outputs exactly: Catchup is set to False, so missed DAG runs will not run automatically. Use airflow backfill to run missed runs manually.

Apache Airflow

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime

with DAG(
    dag_id='example_catchup_dag',
    start_date=datetime(2024, 4, 1),
    schedule_interval='@daily',
    catchup=False
) as dag:
    print_date = BashOperator(
        task_id='print_date',
        bash_command='echo {{ ds }}'
    )

# Your code here to print the message

Need a hint?

Use print() with the exact message inside quotes.