Timetables for complex schedules in Apache Airflow - Step-by-Step Execution

Process Flow - Timetables for complex schedules

Define Timetable Class

↓

Create Timetable Instance

↓

Airflow Scheduler Checks Timetable

↓

Evaluate Current Time Against Timetable

↓

Trigger DAG

↓

Repeat on Schedule

Airflow uses timetable classes to decide when to run DAGs. The scheduler checks the timetable, evaluates if current time matches, then triggers the DAG or waits.

Execution Sample

Apache Airflow

from airflow.timetables.base import Timetable

class MyTimetable(Timetable):
    def next_dagrun_info(self, last_automated, now):
        # logic to define next run
        pass

Defines a custom timetable class that tells Airflow when to schedule the next DAG run.

Process Table

Step	Current Time	Last Run Time	Timetable Check	Next Run Time	Action
1	2024-06-01 08:00	2024-06-01 00:00	Is 08:00 a scheduled time?	2024-06-01 08:00	Trigger DAG
2	2024-06-01 08:05	2024-06-01 08:00	Is 08:05 a scheduled time?	2024-06-01 12:00	Wait
3	2024-06-01 12:00	2024-06-01 08:00	Is 12:00 a scheduled time?	2024-06-01 12:00	Trigger DAG
4	2024-06-01 12:05	2024-06-01 12:00	Is 12:05 a scheduled time?	2024-06-01 16:00	Wait
5	2024-06-01 16:00	2024-06-01 12:00	Is 16:00 a scheduled time?	2024-06-01 16:00	Trigger DAG
6	2024-06-01 16:05	2024-06-01 16:00	Is 16:05 a scheduled time?	2024-06-02 00:00	Wait
7	2024-06-02 00:00	2024-06-01 16:00	Is 00:00 a scheduled time?	2024-06-02 00:00	Trigger DAG
8	2024-06-02 00:05	2024-06-02 00:00	Is 00:05 a scheduled time?	2024-06-02 08:00	Wait
9	2024-06-02 08:00	2024-06-02 00:00	Is 08:00 a scheduled time?	2024-06-02 08:00	Trigger DAG
10	2024-06-02 08:05	2024-06-02 08:00	Is 08:05 a scheduled time?	2024-06-02 12:00	Wait
11	2024-06-02 12:00	2024-06-02 08:00	Is 12:00 a scheduled time?	2024-06-02 12:00	Trigger DAG
Exit	2024-06-02 12:05	2024-06-02 12:00	No more scheduled times now	-	Stop checking until next cycle

💡 No scheduled time matches current time; scheduler waits for next check.

Status Tracker

Variable	Start	After 1	After 2	After 3	After 4	After 5	After 6	After 7	After 8	After 9	After 10	After 11	Final
Current Time	2024-06-01 08:00	2024-06-01 08:05	2024-06-01 12:00	2024-06-01 12:05	2024-06-01 16:00	2024-06-01 16:05	2024-06-02 00:00	2024-06-02 00:05	2024-06-02 08:00	2024-06-02 08:05	2024-06-02 12:00	2024-06-02 12:05	2024-06-02 12:05
Last Run Time	2024-06-01 00:00	2024-06-01 08:00	2024-06-01 08:00	2024-06-01 12:00	2024-06-01 12:00	2024-06-01 16:00	2024-06-01 16:00	2024-06-02 00:00	2024-06-02 00:00	2024-06-02 08:00	2024-06-02 08:00	2024-06-02 12:00	2024-06-02 12:00
Next Run Time	2024-06-01 08:00	2024-06-01 12:00	2024-06-01 12:00	2024-06-01 16:00	2024-06-01 16:00	2024-06-02 00:00	2024-06-02 00:00	2024-06-02 08:00	2024-06-02 08:00	2024-06-02 12:00	2024-06-02 12:00	-	-
Action	Trigger DAG	Wait	Trigger DAG	Wait	Trigger DAG	Wait	Trigger DAG	Wait	Trigger DAG	Wait	Trigger DAG	Stop	Stop

Key Moments - 3 Insights

Why does the scheduler wait at times like 08:05 instead of triggering the DAG?

Because the timetable only triggers DAG runs at exact scheduled times (like 08:00). At 08:05, the condition 'Is current time a scheduled time?' is false, so the scheduler waits for the next scheduled time (see execution_table rows 2 and 4).

How does the timetable know the next run time after a DAG triggers?

After triggering, the timetable calculates the next scheduled time based on its logic (e.g., every 4 hours). This is shown in the 'Next Run Time' column updating after each trigger in the execution_table.

What happens if the current time is not exactly on a scheduled time?

The scheduler does not trigger the DAG and waits. This prevents running DAGs at wrong times and ensures runs happen only at planned schedule points (see execution_table rows with 'Wait' action).

Visual Quiz - 3 Questions

Look at the execution table at Step 3. What is the action taken?

A Stop

B Wait

C Trigger DAG

D Reschedule

💡 Hint

Check the 'Action' column at Step 3 in the execution_table.

At which step does the scheduler first wait instead of triggering the DAG?

A Step 1

B Step 2

C Step 5

D Step 7

💡 Hint

Look for the first 'Wait' action in the execution_table.

If the timetable was changed to trigger every 2 hours instead of 4, how would the 'Next Run Time' change after Step 1?

A Next Run Time would be 10:00

B Next Run Time would be 12:00

C Next Run Time would be 14:00

D Next Run Time would be 08:00

💡 Hint

Consider the current schedule triggers every 4 hours; halving it means next run is 2 hours later.

Concept Snapshot

Airflow Timetables define when DAGs run.
Scheduler checks current time against timetable.
If time matches, DAG triggers; else scheduler waits.
Custom timetables allow complex schedules.
Next run time is calculated after each trigger.

Full Transcript

This visual execution shows how Airflow uses timetables to schedule DAG runs. First, a timetable class is defined with logic to decide next run times. The scheduler checks the current time against this timetable. If the current time matches a scheduled run time, the DAG triggers. Otherwise, the scheduler waits until the next scheduled time. The execution table traces times, last run, next run, and actions step-by-step. Variables like current time and last run time update as the scheduler moves through the day. Key moments clarify why the scheduler waits at non-scheduled times and how next run times are calculated. The quiz tests understanding of when DAGs trigger and how schedule changes affect next run times.

Practice

1. What is the main purpose of a timetable in Apache Airflow?

easy

A. To control when workflows (DAGs) are scheduled to run

B. To store data processed by workflows

C. To define the tasks inside a workflow

D. To monitor the status of running tasks

Solution

1. Step 1: Understand Airflow scheduling

   Airflow uses timetables to decide when to start running workflows (DAGs).

2. Step 2: Differentiate timetable from other components

   Timetables do not store data, define tasks, or monitor tasks; they only schedule runs.

3. Final Answer:

   To control when workflows (DAGs) are scheduled to run -> Option A

4. Quick Check:

   Timetable = schedule control [OK]

Hint: Timetables decide when workflows start running [OK]

Common Mistakes:

• Confusing timetable with task definitions
• Thinking timetable stores data
• Mixing timetable with monitoring features

2. Which of the following is the correct way to import the built-in cron timetable in Airflow 2.5+?

easy

A. import airflow.timetables.Cron as CronTimetable

B. import airflow.timetable.cron as CronTimetable

C. from airflow.schedules import CronTimetable

D. from airflow.timetables.cron import CronTimetable

Solution

1. Step 1: Recall Airflow timetable import paths

   Airflow 2.5+ uses airflow.timetables.cron for cron timetable classes.

2. Step 2: Check syntax correctness

   from airflow.timetables.cron import CronTimetable uses correct module and class name with proper Python import syntax.

3. Final Answer:

   from airflow.timetables.cron import CronTimetable -> Option D

4. Quick Check:

   Correct import path = from airflow.timetables.cron import CronTimetable [OK]

Hint: Use airflow.timetables.cron for cron timetable imports [OK]

Common Mistakes:

• Using wrong module name like 'schedules' instead of 'timetables'
• Incorrect capitalization in module or class names
• Wrong import syntax causing errors

3. Given this Airflow timetable code snippet, what schedule does it represent?

from airflow.timetables.cron import CronTimetable
cron = CronTimetable('0 6 * * 1-5')

medium

A. Runs every day at 6:00 AM

B. Runs every hour from 6 AM to 5 PM

C. Runs at 6:00 AM every weekday (Monday to Friday)

D. Runs only on weekends at 6:00 AM

Solution

1. Step 1: Interpret the cron expression '0 6 * * 1-5'

   The fields mean: minute=0, hour=6, day=any, month=any, weekday=1-5 (Monday to Friday).

2. Step 2: Match cron to schedule meaning

   This means the job runs at 6:00 AM on weekdays only.

3. Final Answer:

   Runs at 6:00 AM every weekday (Monday to Friday) -> Option C

4. Quick Check:

   Cron '0 6 * * 1-5' = Weekday 6 AM [OK]

Hint: Cron weekdays 1-5 means Monday to Friday [OK]

Common Mistakes:

• Assuming it runs every day instead of weekdays only
• Confusing hour and minute fields
• Thinking it runs on weekends

4. You wrote this custom timetable code but Airflow raises an error:

class MyTimetable:
    def __init__(self, cron):
        self.cron = cron

my_tt = MyTimetable('0 0 * * *')

What is the likely cause of the error?

medium

A. Not importing CronTimetable from airflow

B. Missing required methods like 'next_dagrun_info' in the timetable class

C. Using a class instead of a function for timetable

D. Incorrect cron string format

Solution

1. Step 1: Understand Airflow timetable requirements

   Custom timetables must implement specific methods like 'next_dagrun_info' to work.

2. Step 2: Check provided class

   The class only stores cron string but lacks required methods, causing Airflow error.

3. Final Answer:

   Missing required methods like 'next_dagrun_info' in the timetable class -> Option B

4. Quick Check:

   Custom timetable needs required methods [OK]

Hint: Custom timetables must implement Airflow's required methods [OK]

Common Mistakes:

• Assuming any class with cron string works as timetable
• Blaming cron string format without checking methods
• Confusing timetable class with task definitions

5. You want to run a DAG every Monday at 8 AM and also on the 1st day of every month at 9 AM. Which Airflow timetable setup correctly combines these schedules?

hard

A. Use a custom timetable that combines two CronTimetable instances for '0 8 * * 1' and '0 9 1 * *', then merges their schedules

B. Create two DAGs, one with '0 8 * * 1' and another with '0 9 1 * *', then combine their runs manually

C. Use a single CronTimetable with expression '0 8 * * 1,0 9 1 * *'

D. Set schedule_interval='@weekly' and add a sensor to trigger on the 1st day of month

Solution

1. Step 1: Understand the scheduling needs

   We need two different schedules: Mondays at 8 AM and 1st day of month at 9 AM.

2. Step 2: Evaluate options for combining schedules

   A single cron expression cannot represent both times simultaneously. Two DAGs is possible but not ideal. A custom timetable combining two CronTimetable instances is the correct approach.

3. Step 3: Why other options fail

   Use a single CronTimetable with expression '0 8 * * 1,0 9 1 * *' is invalid cron syntax. Create two DAGs, one with '0 8 * * 1' and another with '0 9 1 * *', then combine their runs manually is manual and complex. Set schedule_interval='@weekly' and add a sensor to trigger on the 1st day of month uses sensors which is indirect and less clean.

4. Final Answer:

   Use a custom timetable that combines two CronTimetable instances for '0 8 * * 1' and '0 9 1 * *', then merges their schedules -> Option A

5. Quick Check:

   Combine schedules with custom timetable = Use a custom timetable that combines two CronTimetable instances for '0 8 * * 1' and '0 9 1 * *', then merges their schedules [OK]

Hint: Combine multiple cron schedules with a custom timetable class [OK]

Common Mistakes:

• Trying to put two cron expressions in one string
• Creating multiple DAGs instead of one with combined timetable
• Using sensors instead of proper timetable combination