Timetables for complex schedules in Apache Airflow - Practice Problems & Coding Challenges

Challenge - 5 Problems

💻 Command Output

intermediate

Output of a complex Airflow timetable with multiple intervals

Given this Airflow timetable code, what is the next scheduled run date after 2024-06-01 00:00 UTC?

Apache Airflow

from airflow.timetables.interval import CronDataInterval
from airflow.timetables.base import Timetable
from pendulum import datetime

class CustomTimetable(Timetable):
    def infer_manual_data_interval(self, run_after: datetime):
        return None

    def next_dagrun_info(self, last_automated_data_interval, restriction):
        # Schedule every Monday and Friday at 10:00 UTC
        from airflow.timetables.interval import CronDataInterval
        from pendulum import datetime

        if last_automated_data_interval is None:
            next_start = datetime(2024, 6, 3, 10, 0, 0, tz='UTC')  # Monday
        else:
            last_start = last_automated_data_interval.start
            if last_start.weekday() == 0:  # Monday
                next_start = last_start.add(days=4)  # Friday
            else:
                next_start = last_start.add(days=3)  # Next Monday
        return CronDataInterval(next_start, next_start.add(hours=1))

custom_tt = CustomTimetable()
last_interval = None
restriction = None
next_run = custom_tt.next_dagrun_info(last_interval, restriction).start
print(next_run)

A 2024-06-01T10:00:00+00:00

B 2024-06-03T10:00:00+00:00

C 2024-06-05T10:00:00+00:00

D 2024-06-07T10:00:00+00:00

💡 Hint

Think about the first Monday after June 1st, 2024.

The timetable schedules runs on Mondays and Fridays at 10:00 UTC. Since June 1, 2024 is a Saturday, the next Monday is June 3, 2024 at 10:00 UTC.

🧠 Conceptual

intermediate

Understanding Airflow Timetable restrictions

Which statement best describes the role of the 'restriction' parameter in Airflow's Timetable.next_dagrun_info method?

A It defines the timezone for the scheduled runs.

B It specifies the DAG owner responsible for the run.

C It sets the maximum number of concurrent runs allowed.

D It limits the earliest and latest possible run dates to avoid scheduling outside a window.

💡 Hint

Think about how Airflow avoids scheduling runs too early or too late.

The 'restriction' parameter provides earliest and latest datetime boundaries to ensure the timetable does not schedule runs outside these limits.

🔀 Workflow

advanced

Configuring a DAG with a custom timetable for multiple schedules

You want to create an Airflow DAG that runs every weekday at 9:00 AM and also on the 1st of every month at 12:00 PM. Which approach correctly implements this using a custom timetable?

A Create a custom Timetable class that returns next run dates for weekdays at 9:00 AM and 1st of month at 12:00 PM, alternating between them.

B Use two separate DAGs with standard cron schedules for each timing.

C Set schedule_interval to '0 9 * * 1-5,0 12 1 * *' in the DAG definition.

D Use a single cron expression '0 9,12 1-31 * 1-5' to cover both schedules.

💡 Hint

Think about how to combine two different schedules in one DAG timetable.

A custom Timetable class can programmatically return the next run date for either schedule, alternating or choosing the earliest. Cron expressions cannot combine these two schedules cleanly in one expression.

❓ Troubleshoot

advanced

Diagnosing unexpected DAG run times with a custom timetable

A DAG with a custom timetable runs at unexpected times, including weekends, despite the timetable code specifying only weekdays. What is the most likely cause?

A The Airflow scheduler is ignoring the timetable due to a configuration error.

B The DAG's schedule_interval is overriding the custom timetable.

C The timetable's next_dagrun_info method does not correctly check the last run date and returns incorrect next run dates.

D The timezone setting in Airflow is set to a non-UTC zone causing date shifts.

💡 Hint

Check how the timetable calculates the next run date based on the last run.

If the timetable's logic for calculating the next run date is flawed, it may return dates outside the intended schedule, such as weekends.

✅ Best Practice

expert

Best practice for implementing complex schedules in Airflow

What is the best practice when implementing a complex schedule that requires multiple non-overlapping intervals in Airflow?

A Implement a custom Timetable class that explicitly calculates and returns the next run date considering all intervals.

B Use multiple DAGs each with a simple cron schedule and combine their outputs downstream.

C Use a single cron expression with multiple comma-separated values to cover all intervals.

D Manually trigger DAG runs at required times instead of scheduling.

💡 Hint

Consider maintainability and correctness for complex schedules.

Custom Timetable classes provide precise control and clarity for complex schedules, avoiding the limitations and confusion of combining multiple cron expressions or manual triggers.

Practice

1. What is the main purpose of a timetable in Apache Airflow?

easy

A. To control when workflows (DAGs) are scheduled to run

B. To store data processed by workflows

C. To define the tasks inside a workflow

D. To monitor the status of running tasks

Solution

1. Step 1: Understand Airflow scheduling

   Airflow uses timetables to decide when to start running workflows (DAGs).

2. Step 2: Differentiate timetable from other components

   Timetables do not store data, define tasks, or monitor tasks; they only schedule runs.

3. Final Answer:

   To control when workflows (DAGs) are scheduled to run -> Option A

4. Quick Check:

   Timetable = schedule control [OK]

Hint: Timetables decide when workflows start running [OK]

Common Mistakes:

• Confusing timetable with task definitions
• Thinking timetable stores data
• Mixing timetable with monitoring features

2. Which of the following is the correct way to import the built-in cron timetable in Airflow 2.5+?

easy

A. import airflow.timetables.Cron as CronTimetable

B. import airflow.timetable.cron as CronTimetable

C. from airflow.schedules import CronTimetable

D. from airflow.timetables.cron import CronTimetable

Solution

1. Step 1: Recall Airflow timetable import paths

   Airflow 2.5+ uses airflow.timetables.cron for cron timetable classes.

2. Step 2: Check syntax correctness

   from airflow.timetables.cron import CronTimetable uses correct module and class name with proper Python import syntax.

3. Final Answer:

   from airflow.timetables.cron import CronTimetable -> Option D

4. Quick Check:

   Correct import path = from airflow.timetables.cron import CronTimetable [OK]

Hint: Use airflow.timetables.cron for cron timetable imports [OK]

Common Mistakes:

• Using wrong module name like 'schedules' instead of 'timetables'
• Incorrect capitalization in module or class names
• Wrong import syntax causing errors

3. Given this Airflow timetable code snippet, what schedule does it represent?

from airflow.timetables.cron import CronTimetable
cron = CronTimetable('0 6 * * 1-5')

medium

A. Runs every day at 6:00 AM

B. Runs every hour from 6 AM to 5 PM

C. Runs at 6:00 AM every weekday (Monday to Friday)

D. Runs only on weekends at 6:00 AM

Solution

1. Step 1: Interpret the cron expression '0 6 * * 1-5'

   The fields mean: minute=0, hour=6, day=any, month=any, weekday=1-5 (Monday to Friday).

2. Step 2: Match cron to schedule meaning

   This means the job runs at 6:00 AM on weekdays only.

3. Final Answer:

   Runs at 6:00 AM every weekday (Monday to Friday) -> Option C

4. Quick Check:

   Cron '0 6 * * 1-5' = Weekday 6 AM [OK]

Hint: Cron weekdays 1-5 means Monday to Friday [OK]

Common Mistakes:

• Assuming it runs every day instead of weekdays only
• Confusing hour and minute fields
• Thinking it runs on weekends

4. You wrote this custom timetable code but Airflow raises an error:

class MyTimetable:
    def __init__(self, cron):
        self.cron = cron

my_tt = MyTimetable('0 0 * * *')

What is the likely cause of the error?

medium

A. Not importing CronTimetable from airflow

B. Missing required methods like 'next_dagrun_info' in the timetable class

C. Using a class instead of a function for timetable

D. Incorrect cron string format

Solution

1. Step 1: Understand Airflow timetable requirements

   Custom timetables must implement specific methods like 'next_dagrun_info' to work.

2. Step 2: Check provided class

   The class only stores cron string but lacks required methods, causing Airflow error.

3. Final Answer:

   Missing required methods like 'next_dagrun_info' in the timetable class -> Option B

4. Quick Check:

   Custom timetable needs required methods [OK]

Hint: Custom timetables must implement Airflow's required methods [OK]

Common Mistakes:

• Assuming any class with cron string works as timetable
• Blaming cron string format without checking methods
• Confusing timetable class with task definitions

5. You want to run a DAG every Monday at 8 AM and also on the 1st day of every month at 9 AM. Which Airflow timetable setup correctly combines these schedules?

hard

A. Use a custom timetable that combines two CronTimetable instances for '0 8 * * 1' and '0 9 1 * *', then merges their schedules

B. Create two DAGs, one with '0 8 * * 1' and another with '0 9 1 * *', then combine their runs manually

C. Use a single CronTimetable with expression '0 8 * * 1,0 9 1 * *'

D. Set schedule_interval='@weekly' and add a sensor to trigger on the 1st day of month

Solution

1. Step 1: Understand the scheduling needs

   We need two different schedules: Mondays at 8 AM and 1st day of month at 9 AM.

2. Step 2: Evaluate options for combining schedules

   A single cron expression cannot represent both times simultaneously. Two DAGs is possible but not ideal. A custom timetable combining two CronTimetable instances is the correct approach.

3. Step 3: Why other options fail

   Use a single CronTimetable with expression '0 8 * * 1,0 9 1 * *' is invalid cron syntax. Create two DAGs, one with '0 8 * * 1' and another with '0 9 1 * *', then combine their runs manually is manual and complex. Set schedule_interval='@weekly' and add a sensor to trigger on the 1st day of month uses sensors which is indirect and less clean.

4. Final Answer:

   Use a custom timetable that combines two CronTimetable instances for '0 8 * * 1' and '0 9 1 * *', then merges their schedules -> Option A

5. Quick Check:

   Combine schedules with custom timetable = Use a custom timetable that combines two CronTimetable instances for '0 8 * * 1' and '0 9 1 * *', then merges their schedules [OK]

Hint: Combine multiple cron schedules with a custom timetable class [OK]

Common Mistakes:

• Trying to put two cron expressions in one string
• Creating multiple DAGs instead of one with combined timetable
• Using sensors instead of proper timetable combination