Apache Airflow for ML orchestration in MLOps - Practice Problems & Coding Challenges

Challenge - 5 Problems

💻 Command Output

intermediate

Output of Airflow DAG run command

You run the command airflow dags trigger example_ml_pipeline on your terminal. What is the expected immediate output?

A SyntaxError: invalid syntax in DAG file

B No output, command hangs indefinitely

C Error: DAG 'example_ml_pipeline' not found

D Created <DagRun example_ml_pipeline @ 2024-06-01T12:00:00+00:00: manual__2024-06-01T12:00:00+00:00, externally triggered: True>

💡 Hint

Think about what happens when you trigger a DAG manually.

Triggering a DAG manually creates a DagRun instance with a timestamp and marks it as externally triggered. If the DAG exists and is valid, this message appears.

🧠 Conceptual

intermediate

Purpose of Airflow Sensors in ML pipelines

In an ML pipeline orchestrated by Airflow, what is the main purpose of a Sensor task?

A To wait for an external event or file before continuing the pipeline

B To train the ML model using GPU resources

C To send email notifications after pipeline completion

D To clean up temporary files after pipeline execution

💡 Hint

Sensors are special tasks that pause execution until a condition is met.

Sensors wait for external triggers like file availability or database entries before allowing downstream tasks to run, ensuring dependencies are met.

🔀 Workflow

advanced

Correct order of Airflow tasks for ML model deployment

Arrange the following Airflow tasks in the correct order for a typical ML model deployment pipeline:

A 4,2,1,3

B 4,1,2,3

C 1,4,2,3

D 2,4,1,3

💡 Hint

Think about the logical flow from raw data to deployment.

First, data must be preprocessed, then the model is trained on that data, followed by validation, and finally deployment if validation passes.

❓ Troubleshoot

advanced

Diagnosing Airflow task failure due to missing dependencies

An Airflow task in your ML pipeline fails with the error: TaskInstance failed because upstream task did not complete. What is the most likely cause?

A The Airflow scheduler is down and not triggering tasks

B The task code has a syntax error causing immediate failure

C The upstream task did not run or failed, so this task cannot start

D The DAG file is missing from the dags folder

💡 Hint

Check the status of tasks that run before the failing one.

Airflow enforces task dependencies. If an upstream task fails or is skipped, downstream tasks will not run and report this error.

✅ Best Practice

expert

Best practice for managing ML model versioning in Airflow

Which approach is best for managing multiple versions of ML models in an Airflow orchestration pipeline?

A Store model versions in a version control system and use Airflow variables to select the version

B Overwrite the same model file in storage every time a new model is trained

C Keep all models in a single folder without versioning and rely on timestamps

D Manually rename model files after deployment without tracking in Airflow

💡 Hint

Think about reproducibility and traceability in ML workflows.

Using version control and Airflow variables allows clear tracking and selection of model versions, improving reproducibility and rollback capability.