Pipeline components and DAGs in MLOps - Mini Project: Build & Apply

Building a Simple MLOps Pipeline with Components and DAGs

📖 Scenario: You are working as a data engineer in a team that builds machine learning pipelines. Your task is to create a simple pipeline that has components for data loading, data preprocessing, and model training. These components will be connected in a Directed Acyclic Graph (DAG) to define the order of execution.This project will help you understand how pipeline components and DAGs work in MLOps.

🎯 Goal: Build a simple MLOps pipeline using Python dictionaries to represent components and a list to represent the DAG order. You will create components for data loading, preprocessing, and training, then connect them in a DAG, and finally print the execution order.

📋 What You'll Learn

Create a dictionary called components with keys 'load_data', 'preprocess_data', and 'train_model' each having a string description as value.

Create a list called dag that defines the execution order of the components as 'load_data', 'preprocess_data', 'train_model'.

Use a for loop to iterate over the dag list and print the component name and its description from the components dictionary.

💡 Why This Matters

🌍 Real World

In real MLOps, pipelines are built with components representing tasks like data loading, preprocessing, and training. These tasks are connected in a DAG to control the order of execution.

💼 Career

Understanding pipeline components and DAGs is essential for roles like MLOps engineer, data engineer, and machine learning engineer to automate and manage ML workflows efficiently.

1

Create pipeline components dictionary

Create a dictionary called components with these exact entries: 'load_data': 'Load raw data from source', 'preprocess_data': 'Clean and transform data', and 'train_model': 'Train ML model on processed data'.

MLOps

# Create the components dictionary with exact keys and descriptions
# Your code here

Need a hint?

Use curly braces {} to create a dictionary. Each key is a string like 'load_data' and each value is a string description.

2

Define the DAG execution order

Create a list called dag with the exact order of component names: 'load_data', 'preprocess_data', 'train_model'.

MLOps

components = {
    'load_data': 'Load raw data from source',
    'preprocess_data': 'Clean and transform data',
    'train_model': 'Train ML model on processed data'
}

# Create the dag list with the exact component names in order
# Your code here

Need a hint?

Use square brackets [] to create a list. Put the component names as strings in the correct order.

3

Iterate over DAG and print component info

Use a for loop with variable component to iterate over the dag list. Inside the loop, get the description from components[component] and print the component name and description in this format: "Component: {component}, Description: {description}".

MLOps

components = {
    'load_data': 'Load raw data from source',
    'preprocess_data': 'Clean and transform data',
    'train_model': 'Train ML model on processed data'
}

dag = ['load_data', 'preprocess_data', 'train_model']

# Use a for loop to print each component and its description
# Your code here

Need a hint?

Use a for loop to go through each item in dag. Use f-strings to format the print output.

4

Print the pipeline execution order

Write a print statement to display the text exactly: "Pipeline execution order completed."

MLOps

components = {
    'load_data': 'Load raw data from source',
    'preprocess_data': 'Clean and transform data',
    'train_model': 'Train ML model on processed data'
}

dag = ['load_data', 'preprocess_data', 'train_model']

for component in dag:
    description = components[component]
    print(f"Component: {component}, Description: {description}")

# Print the completion message below
# Your code here

Need a hint?

Use print("Pipeline execution order completed.") exactly to show the final message.