Apache Airflow for ML orchestration in MLOps - Mini Project: Build & Apply

Apache Airflow for ML orchestration

📖 Scenario: You are working as a data engineer in a team that builds machine learning models. You want to automate the steps of your ML workflow using Apache Airflow. This will help your team run the training and evaluation tasks automatically every day without manual work.

🎯 Goal: Build a simple Apache Airflow DAG that orchestrates three ML tasks: data extraction, model training, and model evaluation. You will create the DAG structure, add configuration for scheduling, define the tasks, and finally print the task order to verify the workflow.

📋 What You'll Learn

Create a DAG with the id ml_workflow

Set the DAG schedule interval to run daily at 7 AM

Define three PythonOperator tasks named extract_data, train_model, and evaluate_model

Set task dependencies so that extract_data runs before train_model, and train_model runs before evaluate_model

Print the list of task ids in the order they will run

💡 Why This Matters

🌍 Real World

Automating ML workflows with Apache Airflow helps teams run complex pipelines reliably and on schedule without manual intervention.

💼 Career

Understanding Airflow DAGs and task orchestration is essential for ML engineers and data engineers working in MLOps roles.

1

Create the DAG structure

Import DAG from airflow and create a DAG object called ml_workflow with dag_id='ml_workflow' and start_date=datetime(2024, 1, 1). Import datetime from datetime module.

MLOps

# Your code here

Need a hint?

Use DAG(dag_id='ml_workflow', start_date=datetime(2024, 1, 1)) to create the DAG.

2

Add schedule interval configuration

Add the schedule_interval parameter to the ml_workflow DAG and set it to '0 7 * * *' to run daily at 7 AM.

MLOps

from airflow import DAG
from datetime import datetime

# Add schedule_interval to the DAG below
# Your code here

Need a hint?

Set schedule_interval='0 7 * * *' inside the DAG constructor.

3

Define ML tasks using PythonOperator

Import PythonOperator from airflow.operators.python. Define three tasks named extract_data, train_model, and evaluate_model using PythonOperator. Each task should have a task_id matching its name and a python_callable that is a simple function printing the task name. Assign all tasks to the dag.

MLOps

from airflow import DAG
from datetime import datetime

# Define simple functions for each task
# Define PythonOperator tasks for extract_data, train_model, evaluate_model
# Your code here

Need a hint?

Define simple functions that print messages, then create PythonOperator tasks with matching task_id and python_callable.

4

Set task dependencies and print task order

Set the task order so that extract_data runs before train_model, and train_model runs before evaluate_model using the bitshift operators >>. Then print the list of task ids in the order they will run by accessing dag.topological_sort() and printing each task's task_id.

MLOps

from airflow import DAG
from datetime import datetime
from airflow.operators.python import PythonOperator

# Set task dependencies
# Print the task ids in execution order
# Your code here

Need a hint?

Use extract_data >> train_model >> evaluate_model to set dependencies. Use dag.topological_sort() to get tasks in order.