Catchup and backfill behavior in Apache Airflow - Deep Dive

Overview - Catchup and backfill behavior

What is it?

Catchup and backfill are features in Apache Airflow that control how missed or past scheduled tasks are handled. Catchup means Airflow will run all the past scheduled tasks that were not executed. Backfill is a manual process to run tasks for a specific past date range. These help keep data pipelines consistent even if the system was down or tasks failed.

Why it matters

Without catchup and backfill, missed tasks would never run, causing data gaps and unreliable reports. This can lead to wrong business decisions or broken systems. These features ensure pipelines stay complete and accurate, even after interruptions.

Where it fits

Learners should first understand Airflow basics like DAGs, scheduling, and task execution. After mastering catchup and backfill, they can learn advanced topics like SLA monitoring, retries, and dynamic DAG generation.

Mental Model

Core Idea

Catchup automatically runs all missed scheduled tasks to keep pipelines complete, while backfill lets you manually run tasks for past dates to fix gaps.

Think of it like...

Imagine a TV series you watch weekly. Catchup is like watching all missed episodes automatically when you return after being away. Backfill is like choosing to watch specific old episodes manually to catch up on important storylines.

┌─────────────┐       ┌─────────────┐
│ Scheduled   │──────▶│ Task Runs   │
│ Dates       │       │ (On Time)   │
└─────────────┘       └─────────────┘
       │                     ▲
       │                     │
       ▼                     │
┌─────────────┐       ┌─────────────┐
│ Missed      │──────▶│ Catchup     │
│ Dates       │       │ Runs All    │
└─────────────┘       └─────────────┘

Backfill: Manual trigger to run tasks for any past date range.

Build-Up - 7 Steps

1

Foundation

Understanding Airflow Scheduling Basics

Concept: Learn how Airflow schedules tasks using DAGs and execution dates.

Airflow uses DAGs (Directed Acyclic Graphs) to define workflows. Each DAG has a schedule, like daily or hourly. Airflow triggers tasks at scheduled times, using execution dates to track runs. For example, a daily DAG scheduled at midnight runs tasks for the previous day.

Result

You know how Airflow decides when to run tasks and what execution date means.

Understanding scheduling is key because catchup and backfill depend on how Airflow tracks and triggers past runs.

2

Foundation

What Happens When Tasks Are Missed

Concept: Explore scenarios where scheduled tasks do not run on time.

Tasks can be missed if Airflow is down, paused, or if a DAG is newly added with past schedules. Missed tasks create gaps in data processing. Airflow can either ignore these or try to run them later.

Result

You see why missed tasks happen and why they matter.

Knowing missed tasks exist sets the stage for why catchup and backfill are needed to fix gaps.

3

Intermediate

How Catchup Works Automatically

🤔Before reading on: do you think catchup runs missed tasks immediately or waits for manual trigger? Commit to your answer.

Concept: Catchup runs all missed scheduled tasks automatically when Airflow restarts or DAG is unpaused.

If catchup is enabled (default true), Airflow looks at the last successful run and schedules all missed runs up to the current date. For example, if a daily DAG missed 3 days, catchup runs those 3 days' tasks in order before running today's tasks.

Result

Missed tasks run automatically in sequence to fill gaps.

Understanding catchup helps prevent silent data gaps by ensuring all missed runs are processed without manual work.

4

Intermediate

Backfill: Manual Past Task Runs

🤔Before reading on: do you think backfill runs all past tasks automatically or requires manual command? Commit to your answer.

Concept: Backfill is a manual command to run tasks for a specific past date range, independent of the schedule.

Using the CLI command 'airflow dags backfill', you specify a DAG and date range. Airflow runs tasks for those dates even if they were not scheduled or missed. This is useful for fixing data after errors or adding new DAGs with historical data.

Result

You can manually trigger past task runs to fix or fill data.

Knowing backfill lets you fix specific past data without waiting for catchup or changing schedules.

5

Intermediate

Configuring Catchup Behavior in DAGs

Concept: Learn how to enable or disable catchup per DAG to control automatic past runs.

In the DAG definition, the 'catchup' parameter controls this behavior. Setting catchup=False means Airflow skips all missed runs and only runs the latest scheduled task. This is useful for streaming or real-time pipelines where old data is irrelevant.

Result

You can control whether missed tasks run automatically or not.

Knowing how to configure catchup prevents unwanted heavy processing or data duplication in some pipelines.

6

Advanced

Catchup and Backfill Interaction with Task Dependencies

🤔Before reading on: do you think catchup runs tasks independently or respects dependencies? Commit to your answer.

Concept: Catchup and backfill run tasks respecting DAG dependencies and order, ensuring data correctness.

When catchup or backfill runs multiple past tasks, Airflow executes them in the correct order based on dependencies. For example, if task B depends on task A, Airflow runs A before B for each execution date. This prevents data corruption or errors.

Result

Past tasks run in dependency order, preserving workflow logic.

Understanding this prevents confusion about task failures during catchup or backfill due to dependency issues.

7

Expert

Performance and Pitfalls of Catchup in Production

🤔Before reading on: do you think enabling catchup always improves data quality without downsides? Commit to your answer.

Concept: Catchup can cause performance issues or overload if many past runs accumulate; experts manage this carefully.

In production, enabling catchup on frequently scheduled DAGs that were down for long can trigger thousands of runs, causing resource strain. Experts use catchup=False for streaming DAGs or limit backfill windows. They also monitor task durations and use sensors to avoid overload.

Result

You understand when catchup can harm system stability and how to mitigate it.

Knowing catchup's impact on resources helps design scalable, reliable pipelines avoiding unexpected crashes.

Under the Hood

Airflow tracks DAG runs by execution date in its metadata database. When catchup is enabled, the scheduler queries for all execution dates between the last successful run and the current date. It creates DAG run entries for each missed date and triggers tasks accordingly. Backfill uses a CLI command to insert DAG runs for specified past dates, bypassing the scheduler's normal flow. Task dependencies are resolved per DAG run, ensuring correct execution order.

Why designed this way?

Catchup was designed to ensure data pipelines remain consistent despite downtime or failures, avoiding silent data loss. Backfill provides manual control to fix or fill data gaps on demand. This separation allows automation for normal missed runs and manual intervention for special cases. Alternatives like ignoring missed runs would risk data inconsistency, while always forcing backfill would be too rigid and manual.

┌───────────────┐
│ Scheduler     │
│ Checks DAGs   │
└──────┬────────┘
       │ Queries last successful run
       ▼
┌───────────────┐
│ Metadata DB   │
│ Stores DAGRun │
└──────┬────────┘
       │ Finds missing execution dates
       ▼
┌───────────────┐
│ Catchup Logic │
│ Creates DAGRun│
└──────┬────────┘
       │ Triggers tasks respecting dependencies
       ▼
┌───────────────┐
│ Task Executor │
│ Runs tasks    │
└───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does catchup run missed tasks immediately after Airflow restarts or wait until the next scheduled run? Commit yes or no.

Common Belief: Catchup waits until the next scheduled run to process missed tasks.

Reality: Catchup runs all missed tasks immediately after Airflow restarts or the DAG is unpaused, before running new scheduled tasks.

Why it matters: Believing catchup waits can cause surprise when many tasks suddenly run, potentially overloading the system.

Quick: Does backfill run tasks automatically on schedule or only when manually triggered? Commit your answer.

Common Belief: Backfill runs automatically like catchup whenever tasks are missed.

Reality: Backfill only runs when manually triggered via CLI or API; it does not run automatically.

Why it matters: Assuming automatic backfill can lead to missed data fixes and confusion about pipeline state.

Quick: If catchup is disabled, will Airflow run any missed tasks? Commit yes or no.

Common Belief: Disabling catchup means Airflow will still run missed tasks eventually.

Reality: Disabling catchup means Airflow skips all missed runs and only runs the latest scheduled task.

Why it matters: Misunderstanding this can cause silent data gaps if missed runs are ignored unintentionally.

Quick: Does catchup run tasks ignoring dependencies to speed up processing? Commit yes or no.

Common Belief: Catchup runs missed tasks in parallel ignoring dependencies to catch up faster.

Reality: Catchup respects all task dependencies and runs tasks in the correct order for each execution date.

Why it matters: Ignoring dependencies can cause data corruption or task failures during catchup.

Expert Zone

1

Catchup can cause a 'thundering herd' problem if many DAG runs queue simultaneously, requiring careful resource management.

2

Backfill can be combined with task instance clearing to rerun only failed tasks in a past range, optimizing recovery.

3

Disabling catchup is common in streaming or near-real-time pipelines where only the latest data matters, avoiding unnecessary load.

When NOT to use

Catchup should not be used for pipelines processing real-time or streaming data where old data is irrelevant; instead, set catchup=False. Backfill is not suitable for continuous pipelines and should be used only for manual fixes or historical data loads.

Production Patterns

In production, teams often disable catchup on high-frequency DAGs to avoid overload. They use backfill selectively for data recovery after incidents. Monitoring tools alert when catchup backlog grows too large, triggering manual intervention or DAG pausing.

Connections

Event-driven architecture

Catchup contrasts with event-driven triggers by running tasks based on time schedules rather than events.

Understanding catchup clarifies the difference between time-based and event-based pipeline triggers, helping design hybrid systems.

Database transaction logs

Backfill is similar to replaying transaction logs to restore database state for a past period.

Knowing backfill helps understand how systems recover state by reprocessing past events or data.

Project management backlog

Catchup is like clearing a backlog of unfinished tasks to keep the project on track.

This connection shows how managing missed work in software pipelines parallels managing unfinished tasks in teams.

Common Pitfalls

#1 Leaving catchup enabled on a high-frequency DAG after downtime causes resource overload.

Wrong approach:

dag = DAG('example', schedule_interval='@hourly', catchup=True)

Correct approach:

dag = DAG('example', schedule_interval='@hourly', catchup=False)

Root cause:

Misunderstanding catchup's impact on system load during backlog processing.

#2 Running backfill without clearing previous failed tasks causes duplicate processing.

Wrong approach:

airflow dags backfill example_dag -s 2023-01-01 -e 2023-01-05

Correct approach:

airflow tasks clear example_dag -s 2023-01-01 -e 2023-01-05 && airflow dags backfill example_dag -s 2023-01-01 -e 2023-01-05

Root cause:

Not clearing failed task instances before backfill leads to duplicate or conflicting runs.

#3 Disabling catchup globally without considering DAG-specific needs causes data gaps.

Wrong approach:

airflow.cfg: catchup = False (global setting)

Correct approach:

Set catchup=False per DAG where appropriate, keep default True for others.

Root cause:

Assuming catchup is a global setting and ignoring per-DAG flexibility.

Key Takeaways

Catchup automatically runs all missed scheduled tasks to keep data pipelines complete after downtime.

Backfill is a manual process to run tasks for specific past dates to fix or fill data gaps.

Disabling catchup skips missed runs, useful for real-time pipelines where old data is irrelevant.

Catchup and backfill respect task dependencies, ensuring correct execution order and data integrity.

Misusing catchup can overload systems; experts carefully configure and monitor it in production.