Timetables for complex schedules in Apache Airflow - Commands & Configuration

Introduction

Sometimes you need to run tasks on complicated schedules, like every Monday and Friday at 3 PM or every last day of the month. Airflow timetables help you set these complex schedules clearly and reliably.

When you want to run a task every Monday and Friday at 3 PM exactly.

When you need a job to run on the last day of every month regardless of the date.

When your schedule depends on multiple conditions, like weekdays but excluding holidays.

When you want to combine different schedules into one, like daily at noon plus weekly on Sundays.

When cron expressions become too confusing or limited for your scheduling needs.

Config File - my_dag.py

my_dag.py

from airflow import DAG
from airflow.operators.bash import BashOperator
from airflow.timetables.composite import OrTimetable
from airflow.timetables.cron import CronDataIntervalTimetable
from airflow.utils.dates import days_ago

# Define two cron schedules
cron_monday_friday = CronDataIntervalTimetable('0 15 * * 1,5')  # 3 PM on Monday and Friday
cron_last_day = CronDataIntervalTimetable('0 0 L * *')  # Midnight on last day of month

# Combine schedules with OR logic
combined_timetable = OrTimetable([cron_monday_friday, cron_last_day])

with DAG(
    dag_id='complex_schedule_dag',
    start_date=days_ago(1),
    timetable=combined_timetable,
    catchup=False
) as dag:
    task = BashOperator(
        task_id='print_date',
        bash_command='date'
    )

This DAG file defines two cron-based timetables: one for Mondays and Fridays at 3 PM, and one for the last day of each month at midnight.

These are combined using OrTimetable so the DAG runs when either schedule matches.

The DAG uses this combined timetable to trigger the print_date task accordingly.

Commands

List all DAGs to confirm that the new DAG 'complex_schedule_dag' is recognized by Airflow.

Terminal

airflow dags list

Expected Output

complex_schedule_dag

example_bash_operator

example_python_operator

Manually trigger the DAG to test that it runs correctly with the complex timetable.

Terminal

airflow dags trigger complex_schedule_dag

Expected Output

Created <DagRun complex_schedule_dag @ 2024-06-01T12:00:00+00:00: manual__2024-06-01T12:00:00+00:00, externally triggered: True>

List all tasks in the DAG to verify the task names and structure.

Terminal

airflow tasks list complex_schedule_dag

Expected Output

print_date

Run the 'print_date' task for the DAG run on June 1, 2024, to check the task execution output.

Terminal

airflow tasks test complex_schedule_dag print_date 2024-06-01

Expected Output

[2024-06-01 12:00:00,000] {bash_operator.py:123} INFO - Running command: date

[2024-06-01 12:00:00,100] {bash_operator.py:130} INFO - Fri Jun  1 12:00:00 UTC 2024

Key Concept

If you remember nothing else from this pattern, remember: Airflow timetables let you combine multiple schedules cleanly to run tasks on complex timing rules.

Common Mistakes

Using cron expressions that do not support special characters like 'L' for last day of month.

Standard cron in Airflow does not support 'L', so the schedule won't work as expected.

Use Airflow's CronDataIntervalTimetable which supports extended cron syntax including 'L'.

Not combining multiple timetables and trying to write one complicated cron expression.

Complex schedules become unreadable and error-prone in a single cron expression.

Use OrTimetable to combine simpler cron schedules for clarity and correctness.

Forgetting to set catchup=False when testing complex schedules.

Airflow tries to run all missed intervals, which can flood the scheduler with runs.

Set catchup=False during development to avoid backfilling many runs.

Summary

Define multiple cron schedules using CronDataIntervalTimetable for each timing rule.

Combine these schedules with OrTimetable to create a complex timetable.

Assign the combined timetable to your DAG to run tasks on all specified times.

Practice

1. What is the main purpose of a timetable in Apache Airflow?

easy

A. To control when workflows (DAGs) are scheduled to run

B. To store data processed by workflows

C. To define the tasks inside a workflow

D. To monitor the status of running tasks

Solution

1. Step 1: Understand Airflow scheduling

   Airflow uses timetables to decide when to start running workflows (DAGs).

2. Step 2: Differentiate timetable from other components

   Timetables do not store data, define tasks, or monitor tasks; they only schedule runs.

3. Final Answer:

   To control when workflows (DAGs) are scheduled to run -> Option A

4. Quick Check:

   Timetable = schedule control [OK]

Hint: Timetables decide when workflows start running [OK]

Common Mistakes:

• Confusing timetable with task definitions
• Thinking timetable stores data
• Mixing timetable with monitoring features

2. Which of the following is the correct way to import the built-in cron timetable in Airflow 2.5+?

easy

A. import airflow.timetables.Cron as CronTimetable

B. import airflow.timetable.cron as CronTimetable

C. from airflow.schedules import CronTimetable

D. from airflow.timetables.cron import CronTimetable

Solution

1. Step 1: Recall Airflow timetable import paths

   Airflow 2.5+ uses airflow.timetables.cron for cron timetable classes.

2. Step 2: Check syntax correctness

   from airflow.timetables.cron import CronTimetable uses correct module and class name with proper Python import syntax.

3. Final Answer:

   from airflow.timetables.cron import CronTimetable -> Option D

4. Quick Check:

   Correct import path = from airflow.timetables.cron import CronTimetable [OK]

Hint: Use airflow.timetables.cron for cron timetable imports [OK]

Common Mistakes:

• Using wrong module name like 'schedules' instead of 'timetables'
• Incorrect capitalization in module or class names
• Wrong import syntax causing errors

3. Given this Airflow timetable code snippet, what schedule does it represent?

from airflow.timetables.cron import CronTimetable
cron = CronTimetable('0 6 * * 1-5')

medium

A. Runs every day at 6:00 AM

B. Runs every hour from 6 AM to 5 PM

C. Runs at 6:00 AM every weekday (Monday to Friday)

D. Runs only on weekends at 6:00 AM

Solution

1. Step 1: Interpret the cron expression '0 6 * * 1-5'

   The fields mean: minute=0, hour=6, day=any, month=any, weekday=1-5 (Monday to Friday).

2. Step 2: Match cron to schedule meaning

   This means the job runs at 6:00 AM on weekdays only.

3. Final Answer:

   Runs at 6:00 AM every weekday (Monday to Friday) -> Option C

4. Quick Check:

   Cron '0 6 * * 1-5' = Weekday 6 AM [OK]

Hint: Cron weekdays 1-5 means Monday to Friday [OK]

Common Mistakes:

• Assuming it runs every day instead of weekdays only
• Confusing hour and minute fields
• Thinking it runs on weekends

4. You wrote this custom timetable code but Airflow raises an error:

class MyTimetable:
    def __init__(self, cron):
        self.cron = cron

my_tt = MyTimetable('0 0 * * *')

What is the likely cause of the error?

medium

A. Not importing CronTimetable from airflow

B. Missing required methods like 'next_dagrun_info' in the timetable class

C. Using a class instead of a function for timetable

D. Incorrect cron string format

Solution

1. Step 1: Understand Airflow timetable requirements

   Custom timetables must implement specific methods like 'next_dagrun_info' to work.

2. Step 2: Check provided class

   The class only stores cron string but lacks required methods, causing Airflow error.

3. Final Answer:

   Missing required methods like 'next_dagrun_info' in the timetable class -> Option B

4. Quick Check:

   Custom timetable needs required methods [OK]

Hint: Custom timetables must implement Airflow's required methods [OK]

Common Mistakes:

• Assuming any class with cron string works as timetable
• Blaming cron string format without checking methods
• Confusing timetable class with task definitions

5. You want to run a DAG every Monday at 8 AM and also on the 1st day of every month at 9 AM. Which Airflow timetable setup correctly combines these schedules?

hard

A. Use a custom timetable that combines two CronTimetable instances for '0 8 * * 1' and '0 9 1 * *', then merges their schedules

B. Create two DAGs, one with '0 8 * * 1' and another with '0 9 1 * *', then combine their runs manually

C. Use a single CronTimetable with expression '0 8 * * 1,0 9 1 * *'

D. Set schedule_interval='@weekly' and add a sensor to trigger on the 1st day of month

Solution

1. Step 1: Understand the scheduling needs

   We need two different schedules: Mondays at 8 AM and 1st day of month at 9 AM.

2. Step 2: Evaluate options for combining schedules

   A single cron expression cannot represent both times simultaneously. Two DAGs is possible but not ideal. A custom timetable combining two CronTimetable instances is the correct approach.

3. Step 3: Why other options fail

   Use a single CronTimetable with expression '0 8 * * 1,0 9 1 * *' is invalid cron syntax. Create two DAGs, one with '0 8 * * 1' and another with '0 9 1 * *', then combine their runs manually is manual and complex. Set schedule_interval='@weekly' and add a sensor to trigger on the 1st day of month uses sensors which is indirect and less clean.

4. Final Answer:

   Use a custom timetable that combines two CronTimetable instances for '0 8 * * 1' and '0 9 1 * *', then merges their schedules -> Option A

5. Quick Check:

   Combine schedules with custom timetable = Use a custom timetable that combines two CronTimetable instances for '0 8 * * 1' and '0 9 1 * *', then merges their schedules [OK]

Hint: Combine multiple cron schedules with a custom timetable class [OK]

Common Mistakes:

• Trying to put two cron expressions in one string
• Creating multiple DAGs instead of one with combined timetable
• Using sensors instead of proper timetable combination