Timetables for complex schedules in Apache Airflow - Commands & Configuration

Introduction

Sometimes you need to run tasks on complicated schedules, like every Monday and Friday at 3 PM or every last day of the month. Airflow timetables help you set these complex schedules clearly and reliably.

When you want to run a task every Monday and Friday at 3 PM exactly.

When you need a job to run on the last day of every month regardless of the date.

When your schedule depends on multiple conditions, like weekdays but excluding holidays.

When you want to combine different schedules into one, like daily at noon plus weekly on Sundays.

When cron expressions become too confusing or limited for your scheduling needs.

Config File - my_dag.py

my_dag.py

from airflow import DAG
from airflow.operators.bash import BashOperator
from airflow.timetables.composite import OrTimetable
from airflow.timetables.cron import CronDataIntervalTimetable
from airflow.utils.dates import days_ago

# Define two cron schedules
cron_monday_friday = CronDataIntervalTimetable('0 15 * * 1,5')  # 3 PM on Monday and Friday
cron_last_day = CronDataIntervalTimetable('0 0 L * *')  # Midnight on last day of month

# Combine schedules with OR logic
combined_timetable = OrTimetable([cron_monday_friday, cron_last_day])

with DAG(
    dag_id='complex_schedule_dag',
    start_date=days_ago(1),
    timetable=combined_timetable,
    catchup=False
) as dag:
    task = BashOperator(
        task_id='print_date',
        bash_command='date'
    )

This DAG file defines two cron-based timetables: one for Mondays and Fridays at 3 PM, and one for the last day of each month at midnight.

These are combined using OrTimetable so the DAG runs when either schedule matches.

The DAG uses this combined timetable to trigger the print_date task accordingly.

Commands

List all DAGs to confirm that the new DAG 'complex_schedule_dag' is recognized by Airflow.

Terminal

airflow dags list

Expected Output

complex_schedule_dag

example_bash_operator

example_python_operator

Manually trigger the DAG to test that it runs correctly with the complex timetable.

Terminal

airflow dags trigger complex_schedule_dag

Expected Output

Created <DagRun complex_schedule_dag @ 2024-06-01T12:00:00+00:00: manual__2024-06-01T12:00:00+00:00, externally triggered: True>

List all tasks in the DAG to verify the task names and structure.

Terminal

airflow tasks list complex_schedule_dag

Expected Output

print_date

Run the 'print_date' task for the DAG run on June 1, 2024, to check the task execution output.

Terminal

airflow tasks test complex_schedule_dag print_date 2024-06-01

Expected Output

[2024-06-01 12:00:00,000] {bash_operator.py:123} INFO - Running command: date

[2024-06-01 12:00:00,100] {bash_operator.py:130} INFO - Fri Jun  1 12:00:00 UTC 2024

Key Concept

If you remember nothing else from this pattern, remember: Airflow timetables let you combine multiple schedules cleanly to run tasks on complex timing rules.

Common Mistakes

Using cron expressions that do not support special characters like 'L' for last day of month.

Standard cron in Airflow does not support 'L', so the schedule won't work as expected.

Use Airflow's CronDataIntervalTimetable which supports extended cron syntax including 'L'.

Not combining multiple timetables and trying to write one complicated cron expression.

Complex schedules become unreadable and error-prone in a single cron expression.

Use OrTimetable to combine simpler cron schedules for clarity and correctness.

Forgetting to set catchup=False when testing complex schedules.

Airflow tries to run all missed intervals, which can flood the scheduler with runs.

Set catchup=False during development to avoid backfilling many runs.

Summary

Define multiple cron schedules using CronDataIntervalTimetable for each timing rule.

Combine these schedules with OrTimetable to create a complex timetable.

Assign the combined timetable to your DAG to run tasks on all specified times.