Catchup and backfill behavior in Apache Airflow - Deep Dive

Overview - Catchup and backfill behavior

What is it?

Catchup and backfill are features in Apache Airflow that control how missed or past scheduled tasks are handled. Catchup means Airflow will run all the past scheduled tasks that were not executed. Backfill is a manual process to run tasks for a specific past date range. These help keep data pipelines consistent even if the system was down or tasks failed.

Why it matters

Without catchup and backfill, missed tasks would never run, causing data gaps and unreliable reports. This can lead to wrong business decisions or broken systems. These features ensure pipelines stay complete and accurate, even after interruptions.

Where it fits

Learners should first understand Airflow basics like DAGs, scheduling, and task execution. After mastering catchup and backfill, they can learn advanced topics like SLA monitoring, retries, and dynamic DAG generation.

Mental Model

Core Idea

Catchup automatically runs all missed scheduled tasks to keep pipelines complete, while backfill lets you manually run tasks for past dates to fix gaps.

Think of it like...

Imagine a TV series you watch weekly. Catchup is like watching all missed episodes automatically when you return after being away. Backfill is like choosing to watch specific old episodes manually to catch up on important storylines.

┌─────────────┐       ┌─────────────┐
│ Scheduled   │──────▶│ Task Runs   │
│ Dates       │       │ (On Time)   │
└─────────────┘       └─────────────┘
       │                     ▲
       │                     │
       ▼                     │
┌─────────────┐       ┌─────────────┐
│ Missed      │──────▶│ Catchup     │
│ Dates       │       │ Runs All    │
└─────────────┘       └─────────────┘

Backfill: Manual trigger to run tasks for any past date range.

Build-Up - 7 Steps

1

Foundation

Understanding Airflow Scheduling Basics

Concept: Learn how Airflow schedules tasks using DAGs and execution dates.

Airflow uses DAGs (Directed Acyclic Graphs) to define workflows. Each DAG has a schedule, like daily or hourly. Airflow triggers tasks at scheduled times, using execution dates to track runs. For example, a daily DAG scheduled at midnight runs tasks for the previous day.

Result

You know how Airflow decides when to run tasks and what execution date means.

Understanding scheduling is key because catchup and backfill depend on how Airflow tracks and triggers past runs.

2

Foundation

What Happens When Tasks Are Missed

Concept: Explore scenarios where scheduled tasks do not run on time.

Tasks can be missed if Airflow is down, paused, or if a DAG is newly added with past schedules. Missed tasks create gaps in data processing. Airflow can either ignore these or try to run them later.

Result

You see why missed tasks happen and why they matter.

Knowing missed tasks exist sets the stage for why catchup and backfill are needed to fix gaps.

3

Intermediate

How Catchup Works Automatically

🤔Before reading on: do you think catchup runs missed tasks immediately or waits for manual trigger? Commit to your answer.

Concept: Catchup runs all missed scheduled tasks automatically when Airflow restarts or DAG is unpaused.

If catchup is enabled (default true), Airflow looks at the last successful run and schedules all missed runs up to the current date. For example, if a daily DAG missed 3 days, catchup runs those 3 days' tasks in order before running today's tasks.

Result

Missed tasks run automatically in sequence to fill gaps.

Understanding catchup helps prevent silent data gaps by ensuring all missed runs are processed without manual work.

4

Intermediate

Backfill: Manual Past Task Runs

🤔Before reading on: do you think backfill runs all past tasks automatically or requires manual command? Commit to your answer.

Concept: Backfill is a manual command to run tasks for a specific past date range, independent of the schedule.

Using the CLI command 'airflow dags backfill', you specify a DAG and date range. Airflow runs tasks for those dates even if they were not scheduled or missed. This is useful for fixing data after errors or adding new DAGs with historical data.

Result

You can manually trigger past task runs to fix or fill data.

Knowing backfill lets you fix specific past data without waiting for catchup or changing schedules.

5

Intermediate

Configuring Catchup Behavior in DAGs

Concept: Learn how to enable or disable catchup per DAG to control automatic past runs.

In the DAG definition, the 'catchup' parameter controls this behavior. Setting catchup=False means Airflow skips all missed runs and only runs the latest scheduled task. This is useful for streaming or real-time pipelines where old data is irrelevant.

Result

You can control whether missed tasks run automatically or not.

Knowing how to configure catchup prevents unwanted heavy processing or data duplication in some pipelines.

6

Advanced

Catchup and Backfill Interaction with Task Dependencies

🤔Before reading on: do you think catchup runs tasks independently or respects dependencies? Commit to your answer.

Concept: Catchup and backfill run tasks respecting DAG dependencies and order, ensuring data correctness.

When catchup or backfill runs multiple past tasks, Airflow executes them in the correct order based on dependencies. For example, if task B depends on task A, Airflow runs A before B for each execution date. This prevents data corruption or errors.

Result

Past tasks run in dependency order, preserving workflow logic.

Understanding this prevents confusion about task failures during catchup or backfill due to dependency issues.

7

Expert

Performance and Pitfalls of Catchup in Production

🤔Before reading on: do you think enabling catchup always improves data quality without downsides? Commit to your answer.

Concept: Catchup can cause performance issues or overload if many past runs accumulate; experts manage this carefully.

In production, enabling catchup on frequently scheduled DAGs that were down for long can trigger thousands of runs, causing resource strain. Experts use catchup=False for streaming DAGs or limit backfill windows. They also monitor task durations and use sensors to avoid overload.

Result

You understand when catchup can harm system stability and how to mitigate it.

Knowing catchup's impact on resources helps design scalable, reliable pipelines avoiding unexpected crashes.

Under the Hood

Airflow tracks DAG runs by execution date in its metadata database. When catchup is enabled, the scheduler queries for all execution dates between the last successful run and the current date. It creates DAG run entries for each missed date and triggers tasks accordingly. Backfill uses a CLI command to insert DAG runs for specified past dates, bypassing the scheduler's normal flow. Task dependencies are resolved per DAG run, ensuring correct execution order.

Why designed this way?

Catchup was designed to ensure data pipelines remain consistent despite downtime or failures, avoiding silent data loss. Backfill provides manual control to fix or fill data gaps on demand. This separation allows automation for normal missed runs and manual intervention for special cases. Alternatives like ignoring missed runs would risk data inconsistency, while always forcing backfill would be too rigid and manual.

┌───────────────┐
│ Scheduler     │
│ Checks DAGs   │
└──────┬────────┘
       │ Queries last successful run
       ▼
┌───────────────┐
│ Metadata DB   │
│ Stores DAGRun │
└──────┬────────┘
       │ Finds missing execution dates
       ▼
┌───────────────┐
│ Catchup Logic │
│ Creates DAGRun│
└──────┬────────┘
       │ Triggers tasks respecting dependencies
       ▼
┌───────────────┐
│ Task Executor │
│ Runs tasks    │
└───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does catchup run missed tasks immediately after Airflow restarts or wait until the next scheduled run? Commit yes or no.

Common Belief: Catchup waits until the next scheduled run to process missed tasks.

Reality: Catchup runs all missed tasks immediately after Airflow restarts or the DAG is unpaused, before running new scheduled tasks.

Why it matters: Believing catchup waits can cause surprise when many tasks suddenly run, potentially overloading the system.

Quick: Does backfill run tasks automatically on schedule or only when manually triggered? Commit your answer.

Common Belief: Backfill runs automatically like catchup whenever tasks are missed.

Reality: Backfill only runs when manually triggered via CLI or API; it does not run automatically.

Why it matters: Assuming automatic backfill can lead to missed data fixes and confusion about pipeline state.

Quick: If catchup is disabled, will Airflow run any missed tasks? Commit yes or no.

Common Belief: Disabling catchup means Airflow will still run missed tasks eventually.

Reality: Disabling catchup means Airflow skips all missed runs and only runs the latest scheduled task.

Why it matters: Misunderstanding this can cause silent data gaps if missed runs are ignored unintentionally.

Quick: Does catchup run tasks ignoring dependencies to speed up processing? Commit yes or no.

Common Belief: Catchup runs missed tasks in parallel ignoring dependencies to catch up faster.

Reality: Catchup respects all task dependencies and runs tasks in the correct order for each execution date.

Why it matters: Ignoring dependencies can cause data corruption or task failures during catchup.

Expert Zone

1

Catchup can cause a 'thundering herd' problem if many DAG runs queue simultaneously, requiring careful resource management.

2

Backfill can be combined with task instance clearing to rerun only failed tasks in a past range, optimizing recovery.

3

Disabling catchup is common in streaming or near-real-time pipelines where only the latest data matters, avoiding unnecessary load.

When NOT to use

Catchup should not be used for pipelines processing real-time or streaming data where old data is irrelevant; instead, set catchup=False. Backfill is not suitable for continuous pipelines and should be used only for manual fixes or historical data loads.

Production Patterns

In production, teams often disable catchup on high-frequency DAGs to avoid overload. They use backfill selectively for data recovery after incidents. Monitoring tools alert when catchup backlog grows too large, triggering manual intervention or DAG pausing.

Connections

Event-driven architecture

Catchup contrasts with event-driven triggers by running tasks based on time schedules rather than events.

Understanding catchup clarifies the difference between time-based and event-based pipeline triggers, helping design hybrid systems.

Database transaction logs

Backfill is similar to replaying transaction logs to restore database state for a past period.

Knowing backfill helps understand how systems recover state by reprocessing past events or data.

Project management backlog

Catchup is like clearing a backlog of unfinished tasks to keep the project on track.

This connection shows how managing missed work in software pipelines parallels managing unfinished tasks in teams.

Common Pitfalls

#1 Leaving catchup enabled on a high-frequency DAG after downtime causes resource overload.

Wrong approach:

dag = DAG('example', schedule_interval='@hourly', catchup=True)

Correct approach:

dag = DAG('example', schedule_interval='@hourly', catchup=False)

Root cause:

Misunderstanding catchup's impact on system load during backlog processing.

#2 Running backfill without clearing previous failed tasks causes duplicate processing.

Wrong approach:

airflow dags backfill example_dag -s 2023-01-01 -e 2023-01-05

Correct approach:

airflow tasks clear example_dag -s 2023-01-01 -e 2023-01-05 && airflow dags backfill example_dag -s 2023-01-01 -e 2023-01-05

Root cause:

Not clearing failed task instances before backfill leads to duplicate or conflicting runs.

#3 Disabling catchup globally without considering DAG-specific needs causes data gaps.

Wrong approach:

airflow.cfg: catchup = False (global setting)

Correct approach:

Set catchup=False per DAG where appropriate, keep default True for others.

Root cause:

Assuming catchup is a global setting and ignoring per-DAG flexibility.

Key Takeaways

Catchup automatically runs all missed scheduled tasks to keep data pipelines complete after downtime.

Backfill is a manual process to run tasks for specific past dates to fix or fill data gaps.

Disabling catchup skips missed runs, useful for real-time pipelines where old data is irrelevant.

Catchup and backfill respect task dependencies, ensuring correct execution order and data integrity.

Misusing catchup can overload systems; experts carefully configure and monitor it in production.

Practice

1. What does setting catchup=True do in an Airflow DAG?

easy

A. It manually triggers DAG runs for a specific date range.

B. It disables all scheduled DAG runs.

C. It deletes all past DAG runs.

D. It runs all missed DAG runs automatically to catch up.

Solution

1. Step 1: Understand catchup parameter

   The catchup parameter controls whether Airflow runs missed DAG runs automatically.

2. Step 2: Effect of catchup=True

   When catchup=True, Airflow schedules and runs all DAG runs that were missed since the last run.

3. Final Answer:

   It runs all missed DAG runs automatically to catch up. -> Option D

4. Quick Check:

   catchup=True means run missed DAG runs [OK]

Hint: Catchup=True means Airflow fills missed runs automatically [OK]

Common Mistakes:

• Confusing catchup with manual backfill
• Thinking catchup disables scheduling
• Assuming catchup deletes runs

2. Which of the following is the correct way to disable catchup in an Airflow DAG definition?

easy

A. dag = DAG('my_dag', backfill=False)

B. dag = DAG('my_dag', catchup=True)

C. dag = DAG('my_dag', catchup=False)

D. dag = DAG('my_dag', schedule_interval=None)

Solution

1. Step 1: Identify catchup parameter usage

   The catchup parameter is used to enable or disable catchup behavior in DAGs.

2. Step 2: Disable catchup by setting catchup=False

   Setting catchup=False disables automatic running of missed DAG runs.

3. Final Answer:

   dag = DAG('my_dag', catchup=False) -> Option C

4. Quick Check:

   Disable catchup with catchup=False [OK]

Hint: Use catchup=False to stop automatic missed runs [OK]

Common Mistakes:

• Using backfill parameter instead of catchup
• Setting schedule_interval to None disables scheduling, not catchup
• Setting catchup=True enables, not disables

3. Given a DAG with start_date='2024-04-01', schedule_interval='@daily', and catchup=True, if the DAG was paused and resumed on 2024-04-05, which DAG runs will Airflow execute immediately after resuming?

medium

A. Runs for 2024-04-01 to 2024-04-05

B. Only the run for 2024-04-05

C. No runs will be executed automatically

D. Runs for 2024-04-02 to 2024-04-04 only

Solution

1. Step 1: Understand catchup=True behavior

   With catchup=True, Airflow runs all missed DAG runs from start_date to current date.

2. Step 2: Identify missed runs

   The DAG was paused from 2024-04-01 to 2024-04-04, so runs for 2024-04-01, 02, 03, 04 were missed. On resuming 2024-04-05, Airflow runs all missed plus current.

3. Final Answer:

   Runs for 2024-04-01 to 2024-04-05 -> Option A

4. Quick Check:

   Catchup runs all missed dates including current [OK]

Hint: Catchup runs all missed dates from start_date to now [OK]

Common Mistakes:

• Thinking only current date runs after resume
• Ignoring runs before resume date
• Assuming no runs run automatically after pause

4. You want to run DAG runs manually for the dates 2024-03-01 to 2024-03-05 without enabling catchup. Which command should you use?

medium

A. airflow dags backfill -s 2024-03-01 -e 2024-03-05 my_dag

B. airflow dags trigger my_dag --start-date 2024-03-01 --end-date 2024-03-05

C. airflow dags run my_dag 2024-03-01 to 2024-03-05

D. airflow dags catchup my_dag -s 2024-03-01 -e 2024-03-05

Solution

1. Step 1: Identify manual run command for date range

   The airflow dags backfill command runs DAG runs manually for a specified date range.

2. Step 2: Confirm correct syntax

   Using -s and -e flags with backfill specifies start and end dates for manual runs.

3. Final Answer:

   airflow dags backfill -s 2024-03-01 -e 2024-03-05 my_dag -> Option A

4. Quick Check:

   Backfill runs DAG manually for date range [OK]

Hint: Use airflow dags backfill with -s and -e for manual runs [OK]

Common Mistakes:

• Using trigger command with date range flags (not supported)
• Trying to use catchup command (does not exist)
• Using dags run with invalid date range syntax

5. You have a DAG with catchup=False and a start_date set to 2024-01-01. You want to run all DAG runs from 2024-01-01 to 2024-01-10 manually. What is the best approach to achieve this without enabling catchup?

hard

A. Set catchup=True and wait for Airflow to run all missed DAG runs automatically.

B. Use the backfill command for the date range 2024-01-01 to 2024-01-10.

C. Manually trigger the DAG once on 2024-01-10; Airflow will run all previous runs.

D. Change the start_date to 2024-01-10 and run the DAG normally.

Solution

1. Step 1: Understand catchup=False effect

   With catchup=False, Airflow skips running missed DAG runs automatically.

2. Step 2: Use backfill to run missed DAG runs manually

   To run missed DAG runs without enabling catchup, use the backfill command specifying the date range.

3. Final Answer:

   Use the backfill command for the date range 2024-01-01 to 2024-01-10. -> Option B

4. Quick Check:

   Backfill runs missed DAG runs manually without catchup [OK]

Hint: Backfill runs missed DAG runs manually when catchup is off [OK]

Common Mistakes:

• Enabling catchup when not desired
• Assuming one trigger runs all missed dates
• Changing start_date to skip missed runs