Timetables for complex schedules in Apache Airflow - Practice Problems & Coding Challenges

Challenge - 5 Problems

💻 Command Output

intermediate

Output of a complex Airflow timetable with multiple intervals

Given this Airflow timetable code, what is the next scheduled run date after 2024-06-01 00:00 UTC?

Apache Airflow

from airflow.timetables.interval import CronDataInterval
from airflow.timetables.base import Timetable
from pendulum import datetime

class CustomTimetable(Timetable):
    def infer_manual_data_interval(self, run_after: datetime):
        return None

    def next_dagrun_info(self, last_automated_data_interval, restriction):
        # Schedule every Monday and Friday at 10:00 UTC
        from airflow.timetables.interval import CronDataInterval
        from pendulum import datetime

        if last_automated_data_interval is None:
            next_start = datetime(2024, 6, 3, 10, 0, 0, tz='UTC')  # Monday
        else:
            last_start = last_automated_data_interval.start
            if last_start.weekday() == 0:  # Monday
                next_start = last_start.add(days=4)  # Friday
            else:
                next_start = last_start.add(days=3)  # Next Monday
        return CronDataInterval(next_start, next_start.add(hours=1))

custom_tt = CustomTimetable()
last_interval = None
restriction = None
next_run = custom_tt.next_dagrun_info(last_interval, restriction).start
print(next_run)

A 2024-06-01T10:00:00+00:00

B 2024-06-03T10:00:00+00:00

C 2024-06-05T10:00:00+00:00

D 2024-06-07T10:00:00+00:00

💡 Hint

Think about the first Monday after June 1st, 2024.

The timetable schedules runs on Mondays and Fridays at 10:00 UTC. Since June 1, 2024 is a Saturday, the next Monday is June 3, 2024 at 10:00 UTC.

🧠 Conceptual

intermediate

Understanding Airflow Timetable restrictions

Which statement best describes the role of the 'restriction' parameter in Airflow's Timetable.next_dagrun_info method?

A It defines the timezone for the scheduled runs.

B It specifies the DAG owner responsible for the run.

C It sets the maximum number of concurrent runs allowed.

D It limits the earliest and latest possible run dates to avoid scheduling outside a window.

💡 Hint

Think about how Airflow avoids scheduling runs too early or too late.

The 'restriction' parameter provides earliest and latest datetime boundaries to ensure the timetable does not schedule runs outside these limits.

🔀 Workflow

advanced

Configuring a DAG with a custom timetable for multiple schedules

You want to create an Airflow DAG that runs every weekday at 9:00 AM and also on the 1st of every month at 12:00 PM. Which approach correctly implements this using a custom timetable?

A Create a custom Timetable class that returns next run dates for weekdays at 9:00 AM and 1st of month at 12:00 PM, alternating between them.

B Use two separate DAGs with standard cron schedules for each timing.

C Set schedule_interval to '0 9 * * 1-5,0 12 1 * *' in the DAG definition.

D Use a single cron expression '0 9,12 1-31 * 1-5' to cover both schedules.

💡 Hint

Think about how to combine two different schedules in one DAG timetable.

A custom Timetable class can programmatically return the next run date for either schedule, alternating or choosing the earliest. Cron expressions cannot combine these two schedules cleanly in one expression.

❓ Troubleshoot

advanced

Diagnosing unexpected DAG run times with a custom timetable

A DAG with a custom timetable runs at unexpected times, including weekends, despite the timetable code specifying only weekdays. What is the most likely cause?

A The Airflow scheduler is ignoring the timetable due to a configuration error.

B The DAG's schedule_interval is overriding the custom timetable.

C The timetable's next_dagrun_info method does not correctly check the last run date and returns incorrect next run dates.

D The timezone setting in Airflow is set to a non-UTC zone causing date shifts.

💡 Hint

Check how the timetable calculates the next run date based on the last run.

If the timetable's logic for calculating the next run date is flawed, it may return dates outside the intended schedule, such as weekends.

✅ Best Practice

expert

Best practice for implementing complex schedules in Airflow

What is the best practice when implementing a complex schedule that requires multiple non-overlapping intervals in Airflow?

A Implement a custom Timetable class that explicitly calculates and returns the next run date considering all intervals.

B Use multiple DAGs each with a simple cron schedule and combine their outputs downstream.

C Use a single cron expression with multiple comma-separated values to cover all intervals.

D Manually trigger DAG runs at required times instead of scheduling.

💡 Hint

Consider maintainability and correctness for complex schedules.

Custom Timetable classes provide precise control and clarity for complex schedules, avoiding the limitations and confusion of combining multiple cron expressions or manual triggers.