Catchup and backfill behavior in Apache Airflow - Step-by-Step Execution

Process Flow - Catchup and backfill behavior

DAG Start Date

↓

Scheduler checks DAG runs

↓

Are there missing runs before today?

No→Run only current schedule

Yes↓

Catchup enabled?

No→Run only current schedule

Yes↓

Scheduler creates DAG runs for missing intervals

↓

Runs execute in order

↓

Backfill command?

No→Continue normal scheduling

Yes↓

Backfill creates DAG runs for specified intervals

↓

Runs execute immediately

↓

Backfill complete

This flow shows how Airflow decides to run past DAG runs (catchup) and how manual backfill triggers immediate runs for past intervals.

Execution Sample

Apache Airflow

from airflow import DAG
from airflow.operators.dummy import DummyOperator
from datetime import datetime, timedelta

dag = DAG('example_dag', start_date=datetime(2024,1,1), schedule_interval='@daily', catchup=True)
run_task = DummyOperator(task_id='run', dag=dag)

Defines a DAG starting Jan 1, 2024, scheduled daily with catchup enabled, so missed runs will be created and executed.

Process Table

Step	Date Checked	Catchup Enabled?	Action Taken	DAG Runs Created	Runs Executed
1	2024-01-01	True	Create DAG run for 2024-01-01	2024-01-01	Run for 2024-01-01 executed
2	2024-01-02	True	Create DAG run for 2024-01-02	2024-01-02	Run for 2024-01-02 executed
3	2024-01-03	True	Create DAG run for 2024-01-03	2024-01-03	Run for 2024-01-03 executed
4	2024-01-04	True	No missing runs, run current schedule	None	Run for 2024-01-04 executed
5	Manual Backfill	N/A	Backfill command for 2023-12-29 to 2023-12-31	2023-12-29, 2023-12-30, 2023-12-31	Runs for 2023-12-29 to 2023-12-31 executed immediately
6	2024-01-05	True	No missing runs, run current schedule	None	Run for 2024-01-05 executed

💡 No more missing runs and backfill completed, scheduler continues normal operation.

Status Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4	After Step 5	After Step 6
DAG Runs Created	None	2024-01-01	2024-01-02	2024-01-03	None	2023-12-29, 2023-12-30, 2023-12-31	None
Runs Executed	None	2024-01-01	2024-01-02	2024-01-03	2024-01-04	2023-12-29, 2023-12-30, 2023-12-31	2024-01-05

Key Moments - 3 Insights

Why does Airflow create DAG runs for past dates when catchup is True?

Because catchup=True tells Airflow to create and run DAG runs for all scheduled intervals between the start_date and the current date if they were missed. See execution_table rows 1-3.

What happens if catchup is set to False?

Airflow skips creating DAG runs for past intervals and only runs the DAG for the current scheduled interval. This is shown in execution_table step 4 where no past runs are created.

How is backfill different from catchup?

Backfill is a manual command to run DAG runs for specific past intervals immediately, regardless of catchup setting. See execution_table step 5 where backfill runs execute right away.

Visual Quiz - 3 Questions

Look at the execution_table at step 3. What DAG run date is created and executed?

A 2024-01-04

B 2024-01-03

C 2023-12-31

D 2024-01-05

💡 Hint

Check the 'DAG Runs Created' and 'Runs Executed' columns at step 3.

At which step does the backfill command run DAG runs for past dates?

A Step 5

B Step 4

C Step 2

D Step 6

💡 Hint

Look for the row mentioning 'Manual Backfill' in the 'Date Checked' column.

If catchup was set to False, which step would NOT create DAG runs for past dates?

A Step 5

B Step 1

C Step 3

D Step 4

💡 Hint

Refer to the 'Catchup Enabled?' and 'Action Taken' columns in the execution_table.

Concept Snapshot

Airflow catchup runs missed DAG intervals from start_date to now.
Set catchup=False to skip past runs and run only current.
Backfill is manual, runs past intervals immediately.
Catchup runs are scheduled; backfill runs are manual.
Both create DAG runs for past dates but differ in timing.

Full Transcript

This visual execution shows how Airflow handles catchup and backfill. When catchup is enabled, Airflow creates DAG runs for all missed scheduled intervals from the start date up to the current date, running them in order. If catchup is disabled, Airflow skips these past runs and only runs the current schedule. Backfill is a manual command that creates DAG runs for specified past intervals and runs them immediately, regardless of catchup settings. The execution table traces each step, showing when DAG runs are created and executed, helping beginners understand the difference between automatic catchup and manual backfill.