Why scheduling automates pipeline execution in Apache Airflow - Test Your Understanding

Practice - 5 Tasks

1fill in blank

easy

Complete the code to set the schedule interval for an Airflow DAG.

Apache Airflow

dag = DAG('example_dag', schedule_interval=[1])

Drag options to blanks, or click blank then click option'

A'manual'

B'@daily'

CNone

D'@once'

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'manual' disables scheduling, so the DAG won't run automatically.

The schedule_interval defines how often the DAG runs automatically. '@daily' means it runs once every day.

2fill in blank

medium

Complete the code to import the scheduling module needed for Airflow DAGs.

Apache Airflow

from airflow.utils.[1] import utc

Drag options to blanks, or click blank then click option'

Atimezone

Bdates

Ctimedelta

Doperators

Attempts:

3 left

💡 Hint

Common Mistakes

Confusing timedelta (duration) with timezone (time zone info).

The timezone module helps define the correct time zone for scheduling DAGs.

3fill in blank

hard

Fix the error in the DAG definition to enable scheduling every hour.

Apache Airflow

dag = DAG('hourly_dag', schedule_interval=[1])

Drag options to blanks, or click blank then click option'

A'@daily'

B'hourly'

C'@hourly'

D'* * * * *'

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'hourly' without '@' causes the scheduler to ignore the interval.

The correct string for hourly scheduling in Airflow is '@hourly'.

4fill in blank

hard

Fill both blanks to create a DAG that runs every 15 minutes and starts on January 1, 2024.

Apache Airflow

dag = DAG('quarter_hour_dag', schedule_interval=[1], start_date=[2])

Drag options to blanks, or click blank then click option'

A'*/15 * * * *'

Bdatetime(2024, 1, 1)

Cdatetime(2023, 12, 31)

D'@daily'

Attempts:

3 left

💡 Hint

Common Mistakes

Using a string for start_date instead of a datetime object.

The cron expression '*/15 * * * *' schedules the DAG every 15 minutes. The start_date must be a datetime object representing January 1, 2024.

5fill in blank

hard

Fill all three blanks to define a DAG with a daily schedule, a start date of March 1, 2024, and catchup disabled.

Apache Airflow

dag = DAG('daily_dag', schedule_interval=[1], start_date=[2], catchup=[3])

Drag options to blanks, or click blank then click option'

A'@daily'

Bdatetime(2024, 3, 1)

CFalse

DTrue

Attempts:

3 left

💡 Hint

Common Mistakes

Setting catchup to True causes the DAG to run all past missed intervals.

The DAG runs daily starting March 1, 2024. Setting catchup=False disables running past missed schedules.

Practice

1. What is the main purpose of scheduling in Apache Airflow pipelines?

easy

A. To write code for pipeline tasks

B. To manually trigger pipelines whenever needed

C. To automatically run pipelines at specific times without manual intervention

D. To monitor server hardware status

Solution

1. Step 1: Understand scheduling concept in Airflow

   Scheduling allows pipelines to run automatically at set times, removing the need to start them manually.

2. Step 2: Compare options with scheduling purpose

   Only To automatically run pipelines at specific times without manual intervention describes automatic running at specific times, which matches scheduling's main goal.

3. Final Answer:

   To automatically run pipelines at specific times without manual intervention -> Option C

4. Quick Check:

   Scheduling = automatic timed runs [OK]

Hint: Scheduling means automatic timed runs, not manual starts [OK]

Common Mistakes:

• Confusing scheduling with manual triggering
• Thinking scheduling is about writing code
• Mixing scheduling with hardware monitoring

2. Which Airflow parameter controls when a pipeline runs automatically?

easy

A. schedule_interval

B. start_date

C. task_id

D. retries

Solution

1. Step 1: Identify scheduling parameters in Airflow

   The schedule_interval parameter defines how often the pipeline runs automatically.

2. Step 2: Differentiate from other parameters

   start_date sets when scheduling begins, task_id names tasks, and retries controls retry attempts, not scheduling frequency.

3. Final Answer:

   schedule_interval -> Option A

4. Quick Check:

   schedule_interval = controls run timing [OK]

Hint: schedule_interval sets pipeline run frequency [OK]

Common Mistakes:

• Confusing start_date with schedule_interval
• Using task_id as scheduling control
• Mixing retries with scheduling

3. Given this DAG snippet, when will the pipeline run?

dag = DAG('example', schedule_interval='0 6 * * *', start_date=datetime(2024, 1, 1))

medium

A. Every day at midnight starting January 1, 2024

B. Every hour starting January 1, 2024

C. Only once on January 1, 2024

D. Every day at 6 AM starting January 1, 2024

Solution

1. Step 1: Interpret the schedule_interval cron expression

   The cron '0 6 * * *' means at minute 0, hour 6, every day.

2. Step 2: Understand start_date effect

   The pipeline starts scheduling from January 1, 2024, running daily at 6 AM.

3. Final Answer:

   Every day at 6 AM starting January 1, 2024 -> Option D

4. Quick Check:

   schedule_interval '0 6 * * *' = daily 6 AM [OK]

Hint: Cron '0 6 * * *' means daily at 6 AM [OK]

Common Mistakes:

• Misreading cron as hourly or midnight
• Thinking start_date runs only once
• Ignoring cron syntax meaning

4. You set schedule_interval='@daily' but your pipeline never runs automatically. What is the most likely cause?

medium

A. The DAG's start_date is set in the future

B. The schedule_interval syntax is incorrect

C. The pipeline has no tasks defined

D. The Airflow webserver is down

Solution

1. Step 1: Check start_date impact on scheduling

   If start_date is in the future, Airflow waits until that date to start scheduling.

2. Step 2: Evaluate other options

   @daily is valid syntax, tasks are needed but pipeline won't run if start_date blocks scheduling, and webserver down affects UI but not scheduling itself.

3. Final Answer:

   The DAG's start_date is set in the future -> Option A

4. Quick Check:

   Future start_date blocks scheduling [OK]

Hint: Check start_date is not in the future [OK]

Common Mistakes:

• Assuming schedule_interval syntax is wrong
• Ignoring start_date effect
• Blaming webserver for scheduling issues

5. How does Airflow's scheduling help make pipelines reliable when tasks fail?

hard

A. By ignoring failed tasks and continuing

B. By automatically retrying failed tasks based on retry settings

C. By requiring manual restart of failed tasks

D. By stopping the pipeline permanently on first failure

Solution

1. Step 1: Understand Airflow retry mechanism

   Airflow uses scheduling to retry failed tasks automatically according to retry parameters, improving reliability.

2. Step 2: Compare other options

   Stopping permanently or ignoring failures reduces reliability; manual restarts require extra work, unlike automated retries.

3. Final Answer:

   By automatically retrying failed tasks based on retry settings -> Option B

4. Quick Check:

   Scheduling enables automatic retries [OK]

Hint: Retries automate failure recovery in scheduled pipelines [OK]

Common Mistakes:

• Thinking failures stop pipelines permanently
• Assuming manual intervention is always needed
• Believing failed tasks are ignored