Why DAG design determines pipeline reliability in Apache Airflow - Test Your Understanding

Practice - 5 Tasks

1fill in blank

easy

Complete the code to define a DAG with a daily schedule.

Apache Airflow

dag = DAG('example_dag', schedule_interval=[1], start_date=datetime(2024, 1, 1))

Drag options to blanks, or click blank then click option'

A'@daily'

BNone

C'@weekly'

D'@hourly'

Attempts:

3 left

💡 Hint

Common Mistakes

Using '@hourly' runs the DAG every hour, not daily.

Using None disables scheduling.

The schedule_interval set to '@daily' makes the DAG run once every day.

2fill in blank

medium

Complete the code to set task dependencies so task2 runs after task1.

Apache Airflow

task1 >> [1]

Drag options to blanks, or click blank then click option'

Atask3

Btask4

Ctask1

Dtask2

Attempts:

3 left

💡 Hint

Common Mistakes

Setting task1 after itself causes a circular dependency.

Choosing a task not defined in the DAG.

The >> operator sets the order so task2 runs after task1.

3fill in blank

hard

Fix the error in the DAG definition by completing the missing argument.

Apache Airflow

dag = DAG('my_dag', schedule_interval='@daily', [1]=datetime(2024, 1, 1))

Drag options to blanks, or click blank then click option'

Aend_date

Bstart_date

Cexecution_date

Drun_date

Attempts:

3 left

💡 Hint

Common Mistakes

Using end_date instead of start_date.

Using non-existent arguments like execution_date.

The start_date argument tells Airflow when to start scheduling the DAG.

4fill in blank

hard

Fill both blanks to create a task that runs a Python function using the correct operator and DAG.

Apache Airflow

task = [1](task_id='print_hello', python_callable=print_hello, dag=[2])

Drag options to blanks, or click blank then click option'

APythonOperator

BBashOperator

Cdag

Dmy_dag

Attempts:

3 left

💡 Hint

Common Mistakes

Using BashOperator for Python functions.

Passing the wrong DAG variable.

PythonOperator runs Python functions, and dag is the DAG instance.

5fill in blank

hard

Fill all three blanks to create a dictionary comprehension that maps task IDs to their start dates if the start date is after 2024-01-01.

Apache Airflow

task_start_dates = {task.task_id: task.[1] for task in tasks if task.[2] [3] datetime(2024, 1, 1)}

Drag options to blanks, or click blank then click option'

Astart_date

Bend_date

C>

D<

Attempts:

3 left

💡 Hint

Common Mistakes

Using end_date instead of start_date.

Using the wrong comparison operator.

This comprehension collects start_date of tasks where the start_date is after January 1, 2024.

Practice

1. Why is the design of a DAG important for pipeline reliability in Airflow?

easy

A. Because it controls the order and dependencies of tasks, ensuring correct execution flow.

B. Because it speeds up the execution of tasks by running all tasks in parallel.

C. Because it automatically fixes errors in tasks without user intervention.

D. Because it reduces the number of tasks needed in the pipeline.

Solution

1. Step 1: Understand DAG role in task order

   A DAG (Directed Acyclic Graph) defines how tasks depend on each other and the order they run.

2. Step 2: Connect DAG design to reliability

   Proper design ensures tasks run in the right sequence, preventing errors and delays.

3. Final Answer:

   Because it controls the order and dependencies of tasks, ensuring correct execution flow. -> Option A

4. Quick Check:

   DAG controls task order = B [OK]

Hint: DAG = task order and dependencies for reliability [OK]

Common Mistakes:

• Thinking DAG runs all tasks in parallel always
• Believing DAG fixes task errors automatically
• Assuming fewer tasks means better reliability

2. Which of the following is the correct way to set a task dependency in an Airflow DAG?

easy

A. task1 >> task2

B. task1 << task2

C. task1 + task2

D. task1 & task2

Solution

1. Step 1: Recall Airflow task dependency syntax

   In Airflow, the '>>' operator sets task1 to run before task2.

2. Step 2: Verify other options

   '<<' reverses dependency, '+' and '&' are invalid for dependencies.

3. Final Answer:

   task1 >> task2 -> Option A

4. Quick Check:

   Use '>>' for dependencies = C [OK]

Hint: Use '>>' to set task order in DAG [OK]

Common Mistakes:

• Using '<<' which reverses dependency
• Using '+' or '&' which are invalid operators
• Confusing dependency direction

3. Given the following DAG snippet, what is the execution order of tasks?

task1 >> [task2, task3]
task3 >> task4

medium

A. task1, task2, task3, task4

B. task1, task3, task2, task4

C. task4, task3, task2, task1

D. task1, task2 and task3 in parallel, then task4

Solution

1. Step 1: Analyze dependencies

   task1 runs first, then task2 and task3 run after task1 in parallel.

2. Step 2: Follow downstream tasks

   task4 runs after task3 completes, so task4 waits for task3 only.

3. Final Answer:

   task1, task2 and task3 in parallel, then task4 -> Option D

4. Quick Check:

   Parallel tasks after task1, then task4 = D [OK]

Hint: Square brackets mean parallel tasks after previous [OK]

Common Mistakes:

• Assuming task4 waits for task2 too
• Thinking task2 and task3 run sequentially
• Reading dependencies backwards

4. You notice your Airflow pipeline is stuck and tasks are not running as expected. Which DAG design mistake could cause this?

medium

A. Not setting start_date in DAG

B. Using too many parallel tasks

C. Having circular dependencies between tasks

D. Using task retries

Solution

1. Step 1: Identify common DAG design errors

   Circular dependencies cause tasks to wait on each other forever, blocking execution.

2. Step 2: Evaluate other options

   Too many parallel tasks slow resources but don't block; missing start_date causes scheduling delay; retries help recover.

3. Final Answer:

   Having circular dependencies between tasks -> Option C

4. Quick Check:

   Circular dependencies block DAG = A [OK]

Hint: Circular dependencies cause pipeline to hang [OK]

Common Mistakes:

• Confusing retries with errors
• Thinking missing start_date blocks tasks immediately
• Assuming parallel tasks cause stuck pipeline

5. You want to improve pipeline reliability by redesigning your DAG. Which approach best helps prevent task failures from blocking the entire pipeline?

hard

A. Remove all dependencies so tasks run independently and faster.

B. Add clear dependencies and use trigger rules like 'all_done' to continue despite some failures.

C. Set all tasks to run sequentially to avoid parallel errors.

D. Disable retries to prevent repeated failures.

Solution

1. Step 1: Understand dependency and trigger rules

   Clear dependencies ensure order; 'all_done' trigger rule lets downstream tasks run even if upstream fails.

2. Step 2: Evaluate other options

   Removing dependencies causes chaos; sequential runs slow pipeline; disabling retries reduces fault tolerance.

3. Final Answer:

   Add clear dependencies and use trigger rules like 'all_done' to continue despite some failures. -> Option B

4. Quick Check:

   Use dependencies + trigger rules for reliability = A [OK]

Hint: Use 'all_done' trigger rule to avoid blocking [OK]

Common Mistakes:

• Removing dependencies causing disorder
• Running all tasks sequentially slows pipeline
• Disabling retries reduces fault tolerance