Cron expressions in Airflow - Step-by-Step Execution

Process Flow - Cron expressions in Airflow

Start

↓

Define cron expression

↓

Airflow scheduler parses expression

↓

Calculate next run time

↓

Trigger DAG run at scheduled time

↓

Wait for next schedule

↩Back to Calculate next run time

Airflow reads the cron expression, calculates when to run the task next, triggers the run, then repeats this cycle.

Execution Sample

Apache Airflow

schedule_interval = '0 6 * * *'
# Runs daily at 6:00 AM

This cron expression schedules a DAG to run every day at 6:00 in the morning.

Process Table

Step	Cron Expression Part	Value	Meaning	Next Run Time Calculation
1	Minute	0	At minute 0	Next run at HH:00
2	Hour	6	At hour 6 (6 AM)	Next run at 6:00 AM next day
3	Day of Month	*	Every day of the month	No restriction
4	Month	*	Every month	No restriction
5	Day of Week	*	Every day of the week	No restriction
6	Calculate next run	-	-	Next run scheduled at next 6:00 AM
7	Trigger DAG run	-	-	DAG runs at scheduled time
8	Wait for next schedule	-	-	Cycle repeats daily at 6:00 AM

💡 The scheduler waits until the next 6:00 AM to trigger the DAG again, repeating the cycle.

Status Tracker

Variable	Start	After Step 1	After Step 2	After Step 6	Final
Minute	-	0	0	0	0
Hour	-	-	6	6	6
Day of Month	-	-	*	*	*
Month	-	-	*	*	*
Day of Week	-	-	*	*	*
Next Run Time	None	None	None	Next 6:00 AM	Next 6:00 AM

Key Moments - 3 Insights

Why does the DAG run only once per day even though some parts of the cron expression use '*'?

The '*' means 'every' for that field, but the specific hour and minute (6:00 AM) restrict the run to once daily. See execution_table rows 2 and 3.

What happens if the cron expression is invalid?

Airflow scheduler will fail to parse it and will not schedule the DAG. This is why correct syntax is important before deployment.

How does Airflow know when to trigger the DAG next?

Airflow calculates the next run time based on the cron parts and current time, then triggers the DAG at that time. See execution_table row 6.

Visual Quiz - 3 Questions

Look at the execution table, what is the 'Hour' value at Step 2?

A *

B 0

C 6

D 12

💡 Hint

Check the 'Hour' column value in execution_table row 2.

At which step does Airflow calculate the next run time?

A Step 3

B Step 6

C Step 1

D Step 8

💡 Hint

Look for the step mentioning 'Calculate next run' in the execution_table.

If the minute part changes from '0' to '30', how does the next run time change?

A Runs daily at 6:30 AM

B Runs every minute

C Runs every hour at minute 0

D Runs only once a week

💡 Hint

Minute value controls the exact minute of the hour when the DAG runs, see variable_tracker for 'Minute'.

Concept Snapshot

Cron expressions in Airflow:
- Format: 'minute hour day_of_month month day_of_week'
- Use '*' for 'every' in a field
- Airflow scheduler parses expression to calculate next run
- DAG triggers at calculated times
- Example: '0 6 * * *' runs daily at 6:00 AM

Full Transcript

In Airflow, cron expressions define when a DAG runs. The scheduler reads the expression, breaks it into parts like minute and hour, then calculates the next time to run the DAG. For example, '0 6 * * *' means the DAG runs every day at 6:00 AM. The scheduler waits until that time, triggers the DAG, then repeats the process. Each part of the cron expression controls a time unit, and '*' means every possible value for that unit. Understanding how these parts work together helps schedule tasks precisely.

Practice

1. What does the cron expression 0 12 * * * mean in Airflow scheduling?

easy

A. Run the task every day at 12:00 PM (noon)

B. Run the task every hour at minute 12

C. Run the task every 12 minutes

D. Run the task every day at midnight

Solution

1. Step 1: Understand cron fields

   The expression has five parts: minute, hour, day of month, month, day of week. Here, 0 12 * * * means minute 0, hour 12, any day, any month, any weekday.

2. Step 2: Interpret the time

   Minute 0 and hour 12 means the task runs once daily at 12:00 PM (noon).

3. Final Answer:

   Run the task every day at 12:00 PM (noon) -> Option A

4. Quick Check:

   Minute=0, Hour=12 means noon daily [OK]

Hint: Minute and hour fields show exact run time [OK]

Common Mistakes:

• Confusing hour and minute positions
• Thinking * means a specific time
• Assuming 12 means midnight

2. Which of the following is the correct cron expression to run a task every 15 minutes in Airflow?

easy

A. */15 * * * *

B. 15 * * * *

C. 0 15 * * *

D. * 15 * * *

Solution

1. Step 1: Understand the meaning of '*/15' in cron

   '*/15' in the minute field means every 15 minutes starting at 0.

2. Step 2: Check other options

   15 * * * * runs at minute 15 every hour, 0 15 * * * runs at 3 PM daily, * 15 * * * is invalid because '*' in minute field with 15 in hour is not correct for every 15 minutes.

3. Final Answer:

   */15 * * * * -> Option A

4. Quick Check:

   */15 in minutes means every 15 minutes [OK]

Hint: Use '*/number' in minutes for periodic runs [OK]

Common Mistakes:

• Using fixed minute instead of interval
• Confusing hour and minute fields
• Misplacing the slash '/' syntax

3. Given the cron expression 30 6 * * 1-5, when will the Airflow task run?

medium

A. At 6:30 AM every day

B. At 6:00 AM every weekday

C. At 6:30 PM Monday through Friday

D. At 6:30 AM Monday through Friday

Solution

1. Step 1: Decode each cron field

   Minute=30, Hour=6, Day of month=any, Month=any, Day of week=1-5 (Monday to Friday).

2. Step 2: Interpret the schedule

   The task runs at 6:30 AM on weekdays (Monday to Friday).

3. Final Answer:

   At 6:30 AM Monday through Friday -> Option D

4. Quick Check:

   Day of week 1-5 means weekdays [OK]

Hint: Day of week 1-5 means Monday to Friday [OK]

Common Mistakes:

• Mixing AM and PM hours
• Ignoring day of week range
• Assuming day of month restricts schedule

4. You wrote the cron expression 60 24 * * * in Airflow to run a task daily. What is the problem?

medium

A. It runs every 60 minutes and 24 hours

B. It will run correctly at midnight

C. Minute 60 and hour 24 are invalid values

D. It runs only on the 60th day of the month

Solution

1. Step 1: Check valid ranges for cron fields

   Minute must be 0-59, hour must be 0-23. Here, 60 and 24 are out of range.

2. Step 2: Understand impact of invalid values

   Using invalid values causes syntax errors or the schedule will not run as expected.

3. Final Answer:

   Minute 60 and hour 24 are invalid values -> Option C

4. Quick Check:

   Minute max=59, Hour max=23 [OK]

Hint: Minutes max 59, hours max 23 in cron [OK]

Common Mistakes:

• Using 60 for minutes instead of 0
• Using 24 for hour instead of 0
• Assuming cron accepts out-of-range values

5. You want to schedule an Airflow DAG to run at 11:45 PM on the last day of every month. Which cron expression should you use?

hard

A. 45 23 31 * *

B. 45 23 L * *

C. 45 23 * * 31

D. 45 23 * * L

Solution

1. Step 1: Understand 'L' in cron expressions

   'L' means the last day of the month in the day-of-month field.

2. Step 2: Check each option

   45 23 31 * * runs only on 31st (not all months have 31 days). 45 23 L * * uses 'L' in day-of-month field correctly. 45 23 * * 31 uses 31 in day-of-week field (invalid). 45 23 * * L uses 'L' in day-of-week field (invalid).

3. Final Answer:

   45 23 L * * -> Option B

4. Quick Check:

   'L' in day-of-month means last day [OK]

Hint: Use 'L' in day-of-month for last day of month [OK]

Common Mistakes:

• Using 31 for all months (fails in shorter months)
• Placing 'L' in wrong cron field
• Confusing day-of-week and day-of-month fields