Catchup and backfill behavior in Apache Airflow - Mini Project: Build & Apply

Understanding Catchup and Backfill Behavior in Airflow

📖 Scenario: You are managing scheduled data pipelines using Apache Airflow. Sometimes, your pipelines miss runs due to downtime or errors. You want to understand how Airflow handles these missed runs using catchup and backfill features.

🎯 Goal: Build a simple Airflow DAG that demonstrates how catchup controls whether missed DAG runs are executed automatically, and how to trigger backfill manually.

📋 What You'll Learn

Create an Airflow DAG with a fixed schedule

Set the catchup parameter to control automatic execution of missed runs

Use a BashOperator to simulate a task

Manually trigger backfill using Airflow CLI

Print logs to observe execution dates

💡 Why This Matters

🌍 Real World

In real data engineering projects, pipelines often miss scheduled runs due to system downtime or failures. Understanding catchup and backfill helps ensure data pipelines stay consistent and up to date.

💼 Career

DevOps and data engineers use Airflow catchup and backfill features to manage workflow reliability and data freshness, which are critical for business reporting and analytics.

1

Create a basic Airflow DAG with a BashOperator

Create a DAG named example_catchup_dag with a daily schedule starting from 2024-04-01. Inside the DAG, create a task called print_date using BashOperator that runs the command date.

Apache Airflow

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime

# Your code here

Hint

Use with DAG(...) to define the DAG and include catchup=True to enable catchup.

2

Change the DAG to disable catchup

Modify the DAG to set catchup=False so that Airflow will not automatically run missed DAG runs.

Apache Airflow

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime

with DAG(
    dag_id='example_catchup_dag',
    start_date=datetime(2024, 4, 1),
    schedule_interval='@daily',
    catchup=False  # Change catchup here
) as dag:
    print_date = BashOperator(
        task_id='print_date',
        bash_command='date'
    )

Hint

Set catchup=False inside the DAG arguments.

3

Add a task to print the execution date

Modify the print_date task's bash_command to print the execution date using the Airflow macro {{ ds }}. Use the command echo {{ ds }}.

Apache Airflow

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime

with DAG(
    dag_id='example_catchup_dag',
    start_date=datetime(2024, 4, 1),
    schedule_interval='@daily',
    catchup=False
) as dag:
    print_date = BashOperator(
        task_id='print_date',
        bash_command='echo {{ ds }}'  # Change command to print execution date
    )

Hint

Use bash_command='echo {{ ds }}' to print the execution date.

4

Print a message explaining catchup and backfill

Write a print statement that outputs exactly: Catchup is set to False, so missed DAG runs will not run automatically. Use airflow backfill to run missed runs manually.

Apache Airflow

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime

with DAG(
    dag_id='example_catchup_dag',
    start_date=datetime(2024, 4, 1),
    schedule_interval='@daily',
    catchup=False
) as dag:
    print_date = BashOperator(
        task_id='print_date',
        bash_command='echo {{ ds }}'
    )

# Your code here to print the message

Hint

Use print() with the exact message inside quotes.

Practice

1. What does setting catchup=True do in an Airflow DAG?

easy

A. It manually triggers DAG runs for a specific date range.

B. It disables all scheduled DAG runs.

C. It deletes all past DAG runs.

D. It runs all missed DAG runs automatically to catch up.

Solution

1. Step 1: Understand catchup parameter

   The catchup parameter controls whether Airflow runs missed DAG runs automatically.

2. Step 2: Effect of catchup=True

   When catchup=True, Airflow schedules and runs all DAG runs that were missed since the last run.

3. Final Answer:

   It runs all missed DAG runs automatically to catch up. -> Option D

4. Quick Check:

   catchup=True means run missed DAG runs [OK]

Hint: Catchup=True means Airflow fills missed runs automatically [OK]

Common Mistakes:

• Confusing catchup with manual backfill
• Thinking catchup disables scheduling
• Assuming catchup deletes runs

2. Which of the following is the correct way to disable catchup in an Airflow DAG definition?

easy

A. dag = DAG('my_dag', backfill=False)

B. dag = DAG('my_dag', catchup=True)

C. dag = DAG('my_dag', catchup=False)

D. dag = DAG('my_dag', schedule_interval=None)

Solution

1. Step 1: Identify catchup parameter usage

   The catchup parameter is used to enable or disable catchup behavior in DAGs.

2. Step 2: Disable catchup by setting catchup=False

   Setting catchup=False disables automatic running of missed DAG runs.

3. Final Answer:

   dag = DAG('my_dag', catchup=False) -> Option C

4. Quick Check:

   Disable catchup with catchup=False [OK]

Hint: Use catchup=False to stop automatic missed runs [OK]

Common Mistakes:

• Using backfill parameter instead of catchup
• Setting schedule_interval to None disables scheduling, not catchup
• Setting catchup=True enables, not disables

3. Given a DAG with start_date='2024-04-01', schedule_interval='@daily', and catchup=True, if the DAG was paused and resumed on 2024-04-05, which DAG runs will Airflow execute immediately after resuming?

medium

A. Runs for 2024-04-01 to 2024-04-05

B. Only the run for 2024-04-05

C. No runs will be executed automatically

D. Runs for 2024-04-02 to 2024-04-04 only

Solution

1. Step 1: Understand catchup=True behavior

   With catchup=True, Airflow runs all missed DAG runs from start_date to current date.

2. Step 2: Identify missed runs

   The DAG was paused from 2024-04-01 to 2024-04-04, so runs for 2024-04-01, 02, 03, 04 were missed. On resuming 2024-04-05, Airflow runs all missed plus current.

3. Final Answer:

   Runs for 2024-04-01 to 2024-04-05 -> Option A

4. Quick Check:

   Catchup runs all missed dates including current [OK]

Hint: Catchup runs all missed dates from start_date to now [OK]

Common Mistakes:

• Thinking only current date runs after resume
• Ignoring runs before resume date
• Assuming no runs run automatically after pause

4. You want to run DAG runs manually for the dates 2024-03-01 to 2024-03-05 without enabling catchup. Which command should you use?

medium

A. airflow dags backfill -s 2024-03-01 -e 2024-03-05 my_dag

B. airflow dags trigger my_dag --start-date 2024-03-01 --end-date 2024-03-05

C. airflow dags run my_dag 2024-03-01 to 2024-03-05

D. airflow dags catchup my_dag -s 2024-03-01 -e 2024-03-05

Solution

1. Step 1: Identify manual run command for date range

   The airflow dags backfill command runs DAG runs manually for a specified date range.

2. Step 2: Confirm correct syntax

   Using -s and -e flags with backfill specifies start and end dates for manual runs.

3. Final Answer:

   airflow dags backfill -s 2024-03-01 -e 2024-03-05 my_dag -> Option A

4. Quick Check:

   Backfill runs DAG manually for date range [OK]

Hint: Use airflow dags backfill with -s and -e for manual runs [OK]

Common Mistakes:

• Using trigger command with date range flags (not supported)
• Trying to use catchup command (does not exist)
• Using dags run with invalid date range syntax

5. You have a DAG with catchup=False and a start_date set to 2024-01-01. You want to run all DAG runs from 2024-01-01 to 2024-01-10 manually. What is the best approach to achieve this without enabling catchup?

hard

A. Set catchup=True and wait for Airflow to run all missed DAG runs automatically.

B. Use the backfill command for the date range 2024-01-01 to 2024-01-10.

C. Manually trigger the DAG once on 2024-01-10; Airflow will run all previous runs.

D. Change the start_date to 2024-01-10 and run the DAG normally.

Solution

1. Step 1: Understand catchup=False effect

   With catchup=False, Airflow skips running missed DAG runs automatically.

2. Step 2: Use backfill to run missed DAG runs manually

   To run missed DAG runs without enabling catchup, use the backfill command specifying the date range.

3. Final Answer:

   Use the backfill command for the date range 2024-01-01 to 2024-01-10. -> Option B

4. Quick Check:

   Backfill runs missed DAG runs manually without catchup [OK]

Hint: Backfill runs missed DAG runs manually when catchup is off [OK]

Common Mistakes:

• Enabling catchup when not desired
• Assuming one trigger runs all missed dates
• Changing start_date to skip missed runs