Catchup and backfill behavior in Apache Airflow - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

What happens when catchup is set to True in Airflow?

In Airflow, if you set catchup=True for a DAG, what behavior should you expect when the DAG is first deployed?

A Airflow will only run the DAG for the current date and ignore past schedules.

B Airflow will run all past scheduled DAG runs from the start date until now to catch up.

C Airflow disables scheduling and requires manual trigger for all runs.

D Airflow skips all runs and marks them as success without execution.

💡 Hint

Think about what 'catching up' means in terms of scheduled runs.

When catchup=True, Airflow runs all missed DAG runs between the start_date and the current date to ensure no scheduled runs are skipped.

💻 Command Output

intermediate

Output of backfill command with catchup disabled

Given a DAG with catchup=False, what will be the output when running airflow dags backfill -s 2024-01-01 -e 2024-01-03 my_dag?

Apache Airflow

airflow dags backfill -s 2024-01-01 -e 2024-01-03 my_dag

A Backfill command fails with an error about catchup being disabled.

B No DAG runs are executed because catchup is disabled.

C Only the DAG run for 2024-01-03 is executed.

D Runs DAG for 2024-01-01, 2024-01-02, and 2024-01-03 regardless of catchup setting.

💡 Hint

Remember that backfill manually triggers runs regardless of catchup.

The backfill command manually triggers DAG runs for the specified date range, ignoring the catchup setting.

❓ Troubleshoot

advanced

Why are past DAG runs not executing despite catchup=True?

You have a DAG with catchup=True and a start_date set to 10 days ago. However, no past DAG runs are executing. What is the most likely cause?

A The Airflow scheduler is not running.

B The DAG file has syntax errors preventing execution.

C The DAG's <code>schedule_interval</code> is set to None, so no scheduled runs exist.

D The DAG's <code>max_active_runs</code> is set to 0.

💡 Hint

Check if the DAG has a schedule defined.

If schedule_interval is None, the DAG is considered manual and no scheduled runs are created, so catchup has no effect.

🔀 Workflow

advanced

Order of operations when enabling catchup on an existing DAG

Arrange the steps in the correct order to enable catchup on an existing DAG and ensure past runs are executed.

A 3,1,2,4

B 1,3,2,4

C 3,2,1,4

D 1,2,3,4

💡 Hint

Think about what must be set before enabling catchup and restarting scheduler.

First ensure the start_date is correct, then enable catchup, restart scheduler to apply changes, and finally monitor runs.

✅ Best Practice

expert

Best practice to avoid unintended backfill when enabling catchup

You want to enable catchup=True on a DAG but avoid running all past missed DAG runs immediately. What is the best practice to achieve this?

A Set the DAG's <code>start_date</code> to the current date before enabling catchup.

B Manually trigger a backfill for all past dates before enabling catchup.

C Set <code>max_active_runs</code> to 1 to limit catchup runs.

D Disable the scheduler temporarily while enabling catchup.

💡 Hint

Consider how Airflow decides which past runs to execute.

Setting the start_date to the current date means no past runs exist to catch up on, so enabling catchup won't trigger old runs.