Parameterized pipeline runs in MLOps - Mini Project: Build & Apply

Parameterized Pipeline Runs

📖 Scenario: You work as a data engineer managing machine learning workflows. Your team wants to run the same pipeline multiple times but with different input parameters, like dataset paths and model versions. This helps test different scenarios without changing the pipeline code each time.

🎯 Goal: Build a simple parameterized pipeline script that accepts parameters for dataset_path and model_version, then prints these values. This simulates running a pipeline with different inputs.

📋 What You'll Learn

Create a dictionary called pipeline_params with keys 'dataset_path' and 'model_version' and exact values '/data/train.csv' and 'v1.0'

Add a variable called run_id and set it to 101

Write a function called run_pipeline that takes params and run_id as arguments and prints the parameters in a formatted string

Call run_pipeline with pipeline_params and run_id and print the output

💡 Why This Matters

🌍 Real World

Parameterized pipeline runs allow data teams to test different datasets and model versions without changing the pipeline code, saving time and reducing errors.

💼 Career

Understanding how to pass parameters to pipelines is essential for MLOps engineers and data scientists to automate and scale machine learning workflows.

1

Create pipeline parameters dictionary

Create a dictionary called pipeline_params with these exact entries: 'dataset_path': '/data/train.csv' and 'model_version': 'v1.0'

MLOps

# Create the pipeline_params dictionary with dataset_path and model_version
# Your code here

Need a hint?

Use curly braces {} to create a dictionary with the exact keys and values.

2

Add run ID variable

Add a variable called run_id and set it to the integer 101

MLOps

pipeline_params = {'dataset_path': '/data/train.csv', 'model_version': 'v1.0'}
# Add the run_id variable and set it to 101
# Your code here

Need a hint?

Just assign the number 101 to a variable named run_id.

3

Write the pipeline run function

Write a function called run_pipeline that takes two parameters: params and run_id. Inside, print the message: "Running pipeline with dataset: {dataset_path}, model version: {model_version}, run ID: {run_id}" using the values from params and run_id

MLOps

pipeline_params = {'dataset_path': '/data/train.csv', 'model_version': 'v1.0'}
run_id = 101
# Define the run_pipeline function below
# Your code here

Need a hint?

Use an f-string to format the print message with values from the params dictionary and the run_id variable.

4

Call the pipeline run function

Call the function run_pipeline with pipeline_params and run_id as arguments to print the pipeline run message

MLOps

pipeline_params = {'dataset_path': '/data/train.csv', 'model_version': 'v1.0'}
run_id = 101

def run_pipeline(params, run_id):
    print(f"Running pipeline with dataset: {params['dataset_path']}, model version: {params['model_version']}, run ID: {run_id}")

# Call run_pipeline with pipeline_params and run_id
# Your code here

Need a hint?

Call the function with the exact variable names pipeline_params and run_id.