Apache Airflow for ML orchestration in MLOps - Practice Problems & Coding Challenges

Challenge - 5 Problems

💻 Command Output

intermediate

Output of Airflow DAG run command

You run the command airflow dags trigger example_ml_pipeline on your terminal. What is the expected immediate output?

A SyntaxError: invalid syntax in DAG file

B No output, command hangs indefinitely

C Error: DAG 'example_ml_pipeline' not found

D Created <DagRun example_ml_pipeline @ 2024-06-01T12:00:00+00:00: manual__2024-06-01T12:00:00+00:00, externally triggered: True>

💡 Hint

Think about what happens when you trigger a DAG manually.

Triggering a DAG manually creates a DagRun instance with a timestamp and marks it as externally triggered. If the DAG exists and is valid, this message appears.

🧠 Conceptual

intermediate

Purpose of Airflow Sensors in ML pipelines

In an ML pipeline orchestrated by Airflow, what is the main purpose of a Sensor task?

A To wait for an external event or file before continuing the pipeline

B To train the ML model using GPU resources

C To send email notifications after pipeline completion

D To clean up temporary files after pipeline execution

💡 Hint

Sensors are special tasks that pause execution until a condition is met.

Sensors wait for external triggers like file availability or database entries before allowing downstream tasks to run, ensuring dependencies are met.

🔀 Workflow

advanced

Correct order of Airflow tasks for ML model deployment

Arrange the following Airflow tasks in the correct order for a typical ML model deployment pipeline:

A 4,2,1,3

B 4,1,2,3

C 1,4,2,3

D 2,4,1,3

💡 Hint

Think about the logical flow from raw data to deployment.

First, data must be preprocessed, then the model is trained on that data, followed by validation, and finally deployment if validation passes.

❓ Troubleshoot

advanced

Diagnosing Airflow task failure due to missing dependencies

An Airflow task in your ML pipeline fails with the error: TaskInstance failed because upstream task did not complete. What is the most likely cause?

A The Airflow scheduler is down and not triggering tasks

B The task code has a syntax error causing immediate failure

C The upstream task did not run or failed, so this task cannot start

D The DAG file is missing from the dags folder

💡 Hint

Check the status of tasks that run before the failing one.

Airflow enforces task dependencies. If an upstream task fails or is skipped, downstream tasks will not run and report this error.

✅ Best Practice

expert

Best practice for managing ML model versioning in Airflow

Which approach is best for managing multiple versions of ML models in an Airflow orchestration pipeline?

A Store model versions in a version control system and use Airflow variables to select the version

B Overwrite the same model file in storage every time a new model is trained

C Keep all models in a single folder without versioning and rely on timestamps

D Manually rename model files after deployment without tracking in Airflow

💡 Hint

Think about reproducibility and traceability in ML workflows.

Using version control and Airflow variables allows clear tracking and selection of model versions, improving reproducibility and rollback capability.

Practice

1. What is the main purpose of Apache Airflow in ML orchestration?

easy

A. To store large datasets for ML training

B. To write ML model code in Python

C. To visualize ML model performance metrics

D. To automate and schedule ML workflows as directed tasks

Solution

1. Step 1: Understand Airflow's role

   Apache Airflow is designed to automate workflows by scheduling and running tasks in order.

2. Step 2: Differentiate from other ML tools

   It does not store data, visualize metrics, or write model code but manages task execution.

3. Final Answer:

   To automate and schedule ML workflows as directed tasks -> Option D

4. Quick Check:

   Airflow = workflow automation [OK]

Hint: Airflow schedules tasks, not data or model code [OK]

Common Mistakes:

• Confusing Airflow with data storage tools
• Thinking Airflow writes ML model code
• Assuming Airflow visualizes model metrics

2. Which of the following is the correct way to define a DAG in Apache Airflow using Python?

easy

A. dag = DAG('my_dag', run_every='daily')

B. dag = DAG('my_dag', schedule_interval='@daily')

C. dag = DAG('my_dag', interval='daily')

D. dag = DAG('my_dag', schedule='daily')

Solution

1. Step 1: Recall DAG initialization syntax

   The correct parameter to set schedule is schedule_interval, not run_every, interval, or schedule.

2. Step 2: Verify the example

   dag = DAG('my_dag', schedule_interval='@daily') is the standard syntax to schedule daily runs.

3. Final Answer:

   dag = DAG('my_dag', schedule_interval='@daily') -> Option B

4. Quick Check:

   Use schedule_interval to set DAG timing [OK]

Hint: Use schedule_interval to set DAG timing [OK]

Common Mistakes:

• Using incorrect parameter names like run_every
• Confusing schedule_interval with schedule
• Forgetting to use quotes around '@daily'

3. Given the following Airflow DAG snippet, what will be the order of task execution?

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime

def task_a():
    print('Task A')

def task_b():
    print('Task B')

def task_c():
    print('Task C')

dag = DAG('example_dag', start_date=datetime(2024, 1, 1), schedule_interval='@once')

t1 = PythonOperator(task_id='a', python_callable=task_a, dag=dag)
t2 = PythonOperator(task_id='b', python_callable=task_b, dag=dag)
t3 = PythonOperator(task_id='c', python_callable=task_c, dag=dag)

t1 >> t2 >> t3

medium

A. Task A, then Task B, then Task C

B. Task C, then Task B, then Task A

C. Task A, Task B, and Task C run in parallel

D. Task B, then Task A, then Task C

Solution

1. Step 1: Understand task dependencies

   The operator chaining t1 >> t2 >> t3 means t1 runs first, then t2, then t3.

2. Step 2: Confirm execution order

   Tasks print in order: Task A, Task B, Task C.

3. Final Answer:

   Task A, then Task B, then Task C -> Option A

4. Quick Check:

   Operator chaining sets task order [OK]

Hint: >> means run left task before right task [OK]

Common Mistakes:

• Assuming tasks run in parallel without dependencies
• Misreading the >> operator order
• Confusing task IDs with execution order

4. You wrote this Airflow DAG code but get an error: TypeError: DAG.__init__() got an unexpected keyword argument 'start'
What is the likely cause?

dag = DAG('my_dag', start='2024-01-01', schedule_interval='@daily')

medium

A. The parameter should be start_date, not start

B. The schedule_interval value '@daily' is invalid

C. DAG name cannot be 'my_dag'

D. Missing import for datetime module

Solution

1. Step 1: Identify incorrect parameter

   The error says start is unexpected; Airflow expects start_date.

2. Step 2: Confirm correct parameter usage

   Replacing start with start_date fixes the error.

3. Final Answer:

   The parameter should be start_date, not start -> Option A

4. Quick Check:

   Use start_date, not start [OK]

Hint: Use start_date, not start, for DAG start time [OK]

Common Mistakes:

• Using 'start' instead of 'start_date'
• Assuming '@daily' is invalid schedule
• Ignoring error message details

5. You want to create an Airflow DAG that runs an ML training task only if data preprocessing succeeded. Which Airflow feature should you use to enforce this dependency?

hard

A. Schedule both tasks to run at the same time

B. Use Airflow Variables to store task status

C. Set task dependencies using >> operator between preprocessing and training tasks

D. Write a single Python function combining both tasks

Solution

1. Step 1: Understand task dependency in Airflow

   Airflow uses task dependencies to control execution order, ensuring one task runs after another succeeds.

2. Step 2: Apply dependency operator

   Using the >> operator sets the training task to run only after preprocessing completes successfully.

3. Final Answer:

   Set task dependencies using >> operator between preprocessing and training tasks -> Option C

4. Quick Check:

   Use >> to enforce task order [OK]

Hint: Use >> to link tasks in order [OK]

Common Mistakes:

• Thinking Variables control task order
• Scheduling tasks simultaneously without dependencies
• Combining tasks loses modularity and control