Cron expressions in Airflow - Mini Project: Build & Apply

Cron expressions in Airflow

📖 Scenario: You are setting up scheduled tasks in Apache Airflow to automate data workflows. You want to learn how to use cron expressions to control when your tasks run.

🎯 Goal: Build a simple Airflow DAG that uses a cron expression to schedule a task to run every day at 3:30 AM.

📋 What You'll Learn

Create a DAG with the exact id daily_report

Use a cron expression 30 3 * * * for the schedule_interval

Define a PythonOperator task with the id print_hello

The task should run a Python function that prints 'Hello Airflow'

Print the DAG's schedule_interval at the end

💡 Why This Matters

🌍 Real World

Scheduling workflows to run automatically at specific times is common in data engineering and DevOps. Airflow uses cron expressions to control these schedules.

💼 Career

Understanding cron expressions in Airflow is essential for roles like Data Engineer, DevOps Engineer, and Automation Specialist to automate and manage workflows reliably.

1

Create the Airflow DAG with the correct id

Import DAG from airflow and create a DAG object called dag with the id 'daily_report'. Use start_date as datetime(2024, 1, 1) and catchup=false.

Apache Airflow

# Import DAG and datetime
# Create DAG with dag_id 'daily_report', start_date datetime(2024, 1, 1), catchup=false
# Your code here

Hint

Use DAG(dag_id='daily_report', start_date=datetime(2024, 1, 1), catchup=false) to create the DAG.

2

Add the cron expression schedule_interval

Add the parameter schedule_interval='30 3 * * *' to the existing DAG creation line to schedule the DAG to run daily at 3:30 AM.

Apache Airflow

from airflow import DAG
from datetime import datetime

# Add schedule_interval='30 3 * * *' to the DAG
# Your code here

Hint

Add schedule_interval='30 3 * * *' inside the DAG() parentheses.

3

Define a PythonOperator task that prints 'Hello Airflow'

Import PythonOperator from airflow.operators.python. Define a Python function called print_hello that prints 'Hello Airflow'. Then create a PythonOperator task with task_id='print_hello', python_callable=print_hello, and assign it to the dag.

Apache Airflow

from airflow import DAG
from datetime import datetime

# Import PythonOperator
# Define function print_hello that prints 'Hello Airflow'
# Create PythonOperator task with task_id 'print_hello' and python_callable print_hello
# Your code here

Hint

Define the function first, then create the PythonOperator with the correct parameters.

4

Print the DAG's schedule_interval

Write a print statement to display the schedule_interval of the dag object.

Apache Airflow

from airflow import DAG
from datetime import datetime
from airflow.operators.python import PythonOperator

# Print the schedule_interval of the dag
# Your code here

Hint

Use print(dag.schedule_interval) to show the cron expression.

Practice

1. What does the cron expression 0 12 * * * mean in Airflow scheduling?

easy

A. Run the task every day at 12:00 PM (noon)

B. Run the task every hour at minute 12

C. Run the task every 12 minutes

D. Run the task every day at midnight

Solution

1. Step 1: Understand cron fields

   The expression has five parts: minute, hour, day of month, month, day of week. Here, 0 12 * * * means minute 0, hour 12, any day, any month, any weekday.

2. Step 2: Interpret the time

   Minute 0 and hour 12 means the task runs once daily at 12:00 PM (noon).

3. Final Answer:

   Run the task every day at 12:00 PM (noon) -> Option A

4. Quick Check:

   Minute=0, Hour=12 means noon daily [OK]

Hint: Minute and hour fields show exact run time [OK]

Common Mistakes:

• Confusing hour and minute positions
• Thinking * means a specific time
• Assuming 12 means midnight

2. Which of the following is the correct cron expression to run a task every 15 minutes in Airflow?

easy

A. */15 * * * *

B. 15 * * * *

C. 0 15 * * *

D. * 15 * * *

Solution

1. Step 1: Understand the meaning of '*/15' in cron

   '*/15' in the minute field means every 15 minutes starting at 0.

2. Step 2: Check other options

   15 * * * * runs at minute 15 every hour, 0 15 * * * runs at 3 PM daily, * 15 * * * is invalid because '*' in minute field with 15 in hour is not correct for every 15 minutes.

3. Final Answer:

   */15 * * * * -> Option A

4. Quick Check:

   */15 in minutes means every 15 minutes [OK]

Hint: Use '*/number' in minutes for periodic runs [OK]

Common Mistakes:

• Using fixed minute instead of interval
• Confusing hour and minute fields
• Misplacing the slash '/' syntax

3. Given the cron expression 30 6 * * 1-5, when will the Airflow task run?

medium

A. At 6:30 AM every day

B. At 6:00 AM every weekday

C. At 6:30 PM Monday through Friday

D. At 6:30 AM Monday through Friday

Solution

1. Step 1: Decode each cron field

   Minute=30, Hour=6, Day of month=any, Month=any, Day of week=1-5 (Monday to Friday).

2. Step 2: Interpret the schedule

   The task runs at 6:30 AM on weekdays (Monday to Friday).

3. Final Answer:

   At 6:30 AM Monday through Friday -> Option D

4. Quick Check:

   Day of week 1-5 means weekdays [OK]

Hint: Day of week 1-5 means Monday to Friday [OK]

Common Mistakes:

• Mixing AM and PM hours
• Ignoring day of week range
• Assuming day of month restricts schedule

4. You wrote the cron expression 60 24 * * * in Airflow to run a task daily. What is the problem?

medium

A. It runs every 60 minutes and 24 hours

B. It will run correctly at midnight

C. Minute 60 and hour 24 are invalid values

D. It runs only on the 60th day of the month

Solution

1. Step 1: Check valid ranges for cron fields

   Minute must be 0-59, hour must be 0-23. Here, 60 and 24 are out of range.

2. Step 2: Understand impact of invalid values

   Using invalid values causes syntax errors or the schedule will not run as expected.

3. Final Answer:

   Minute 60 and hour 24 are invalid values -> Option C

4. Quick Check:

   Minute max=59, Hour max=23 [OK]

Hint: Minutes max 59, hours max 23 in cron [OK]

Common Mistakes:

• Using 60 for minutes instead of 0
• Using 24 for hour instead of 0
• Assuming cron accepts out-of-range values

5. You want to schedule an Airflow DAG to run at 11:45 PM on the last day of every month. Which cron expression should you use?

hard

A. 45 23 31 * *

B. 45 23 L * *

C. 45 23 * * 31

D. 45 23 * * L

Solution

1. Step 1: Understand 'L' in cron expressions

   'L' means the last day of the month in the day-of-month field.

2. Step 2: Check each option

   45 23 31 * * runs only on 31st (not all months have 31 days). 45 23 L * * uses 'L' in day-of-month field correctly. 45 23 * * 31 uses 31 in day-of-week field (invalid). 45 23 * * L uses 'L' in day-of-week field (invalid).

3. Final Answer:

   45 23 L * * -> Option B

4. Quick Check:

   'L' in day-of-month means last day [OK]

Hint: Use 'L' in day-of-month for last day of month [OK]

Common Mistakes:

• Using 31 for all months (fails in shorter months)
• Placing 'L' in wrong cron field
• Confusing day-of-week and day-of-month fields