Cron expressions in Airflow - Commands & Configuration

Introduction

Scheduling tasks to run automatically at specific times can be tricky. Cron expressions help by letting you set exact schedules in Airflow, so your workflows run when you want without manual effort.

When you want a task to run every day at 3 AM to process daily reports.

When you need a job to run every Monday at 9 AM to start weekly data backups.

When you want to run a task every 15 minutes to check for new data arrivals.

When you want to run a task at midnight on the first day of every month for monthly billing.

When you want to schedule a task to run every hour during business hours only.

Config File - my_dag.py

my_dag.py

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime

with DAG(
    dag_id='cron_expression_example',
    start_date=datetime(2024, 1, 1),
    schedule_interval='0 3 * * *',  # Runs daily at 3 AM
    catchup=False
) as dag:
    task = BashOperator(
        task_id='print_date',
        bash_command='date'
    )

This DAG file defines a simple Airflow workflow.

dag_id names the workflow.

start_date sets when the schedule starts.

schedule_interval uses a cron expression to run the task daily at 3 AM.

catchup=False means it won't run missed schedules from the past.

The BashOperator runs the date command as a task.

Commands

Lists all DAGs currently available in Airflow to verify your DAG is recognized.

Terminal

airflow dags list

Expected Output

dag_id              | filepath

cron_expression_example | /usr/local/airflow/dags/my_dag.py

Manually triggers the DAG to run immediately, useful for testing your schedule and tasks.

Terminal

airflow dags trigger cron_expression_example

Expected Output

Created <DagRun cron_expression_example @ 2024-06-01T12:00:00+00:00: manual__2024-06-01T12:00:00+00:00, externally triggered: True>

Lists all tasks inside the DAG to confirm the task names and structure.

Terminal

airflow tasks list cron_expression_example

Expected Output

print_date

Runs the 'print_date' task for the given date without scheduling, to check task behavior.

Terminal

airflow tasks test cron_expression_example print_date 2024-06-01

Expected Output

[2024-06-01 12:00:00,000] {bash.py:123} INFO - Running command: date

[2024-06-01 12:00:00,100] {bash.py:130} INFO - Output:

Wed Jun  1 12:00:00 UTC 2024

[2024-06-01 12:00:00,200] {taskinstance.py:123} INFO - Task succeeded

Key Concept

If you remember nothing else from this pattern, remember: cron expressions let you precisely schedule Airflow tasks by defining minute, hour, day, month, and weekday fields.

Common Mistakes

Using a cron expression with wrong field order or missing fields.

Airflow expects 5 fields in the order: minute, hour, day of month, month, day of week. Wrong order causes scheduling errors or no runs.

Always write cron expressions as 'minute hour day month weekday', for example '0 3 * * *' for daily at 3 AM.

Setting start_date in the future or forgetting catchup=False.

If start_date is in the future, the DAG won't run until that date. If catchup=True (default), Airflow tries to run all missed schedules, which can cause many backlogs.

Set start_date to a past date and use catchup=False to avoid unexpected backlogs.

Using '@daily' or '@hourly' without understanding they are shortcuts.

Shortcuts work but can be confusing if you want custom times. They also may not fit all scheduling needs.

Use full cron expressions for precise control, especially for non-standard schedules.

Summary

Define your Airflow DAG with a cron expression in the schedule parameter to control when tasks run.

Use airflow CLI commands to list DAGs, trigger runs, list tasks, and test tasks for validation.

Remember the cron format is minute, hour, day of month, month, and day of week for correct scheduling.

Practice

1. What does the cron expression 0 12 * * * mean in Airflow scheduling?

easy

A. Run the task every day at 12:00 PM (noon)

B. Run the task every hour at minute 12

C. Run the task every 12 minutes

D. Run the task every day at midnight

Solution

1. Step 1: Understand cron fields

   The expression has five parts: minute, hour, day of month, month, day of week. Here, 0 12 * * * means minute 0, hour 12, any day, any month, any weekday.

2. Step 2: Interpret the time

   Minute 0 and hour 12 means the task runs once daily at 12:00 PM (noon).

3. Final Answer:

   Run the task every day at 12:00 PM (noon) -> Option A

4. Quick Check:

   Minute=0, Hour=12 means noon daily [OK]

Hint: Minute and hour fields show exact run time [OK]

Common Mistakes:

• Confusing hour and minute positions
• Thinking * means a specific time
• Assuming 12 means midnight

2. Which of the following is the correct cron expression to run a task every 15 minutes in Airflow?

easy

A. */15 * * * *

B. 15 * * * *

C. 0 15 * * *

D. * 15 * * *

Solution

1. Step 1: Understand the meaning of '*/15' in cron

   '*/15' in the minute field means every 15 minutes starting at 0.

2. Step 2: Check other options

   15 * * * * runs at minute 15 every hour, 0 15 * * * runs at 3 PM daily, * 15 * * * is invalid because '*' in minute field with 15 in hour is not correct for every 15 minutes.

3. Final Answer:

   */15 * * * * -> Option A

4. Quick Check:

   */15 in minutes means every 15 minutes [OK]

Hint: Use '*/number' in minutes for periodic runs [OK]

Common Mistakes:

• Using fixed minute instead of interval
• Confusing hour and minute fields
• Misplacing the slash '/' syntax

3. Given the cron expression 30 6 * * 1-5, when will the Airflow task run?

medium

A. At 6:30 AM every day

B. At 6:00 AM every weekday

C. At 6:30 PM Monday through Friday

D. At 6:30 AM Monday through Friday

Solution

1. Step 1: Decode each cron field

   Minute=30, Hour=6, Day of month=any, Month=any, Day of week=1-5 (Monday to Friday).

2. Step 2: Interpret the schedule

   The task runs at 6:30 AM on weekdays (Monday to Friday).

3. Final Answer:

   At 6:30 AM Monday through Friday -> Option D

4. Quick Check:

   Day of week 1-5 means weekdays [OK]

Hint: Day of week 1-5 means Monday to Friday [OK]

Common Mistakes:

• Mixing AM and PM hours
• Ignoring day of week range
• Assuming day of month restricts schedule

4. You wrote the cron expression 60 24 * * * in Airflow to run a task daily. What is the problem?

medium

A. It runs every 60 minutes and 24 hours

B. It will run correctly at midnight

C. Minute 60 and hour 24 are invalid values

D. It runs only on the 60th day of the month

Solution

1. Step 1: Check valid ranges for cron fields

   Minute must be 0-59, hour must be 0-23. Here, 60 and 24 are out of range.

2. Step 2: Understand impact of invalid values

   Using invalid values causes syntax errors or the schedule will not run as expected.

3. Final Answer:

   Minute 60 and hour 24 are invalid values -> Option C

4. Quick Check:

   Minute max=59, Hour max=23 [OK]

Hint: Minutes max 59, hours max 23 in cron [OK]

Common Mistakes:

• Using 60 for minutes instead of 0
• Using 24 for hour instead of 0
• Assuming cron accepts out-of-range values

5. You want to schedule an Airflow DAG to run at 11:45 PM on the last day of every month. Which cron expression should you use?

hard

A. 45 23 31 * *

B. 45 23 L * *

C. 45 23 * * 31

D. 45 23 * * L

Solution

1. Step 1: Understand 'L' in cron expressions

   'L' means the last day of the month in the day-of-month field.

2. Step 2: Check each option

   45 23 31 * * runs only on 31st (not all months have 31 days). 45 23 L * * uses 'L' in day-of-month field correctly. 45 23 * * 31 uses 31 in day-of-week field (invalid). 45 23 * * L uses 'L' in day-of-week field (invalid).

3. Final Answer:

   45 23 L * * -> Option B

4. Quick Check:

   'L' in day-of-month means last day [OK]

Hint: Use 'L' in day-of-month for last day of month [OK]

Common Mistakes:

• Using 31 for all months (fails in shorter months)
• Placing 'L' in wrong cron field
• Confusing day-of-week and day-of-month fields