Parameterized pipeline runs in MLOps - Mini Project: Build & Apply

Parameterized Pipeline Runs

📖 Scenario: You work as a data engineer managing machine learning workflows. Your team wants to run the same pipeline multiple times but with different input parameters, like dataset paths and model versions. This helps test different scenarios without changing the pipeline code each time.

🎯 Goal: Build a simple parameterized pipeline script that accepts parameters for dataset_path and model_version, then prints these values. This simulates running a pipeline with different inputs.

📋 What You'll Learn

Create a dictionary called pipeline_params with keys 'dataset_path' and 'model_version' and exact values '/data/train.csv' and 'v1.0'

Add a variable called run_id and set it to 101

Write a function called run_pipeline that takes params and run_id as arguments and prints the parameters in a formatted string

Call run_pipeline with pipeline_params and run_id and print the output

💡 Why This Matters

🌍 Real World

Parameterized pipeline runs allow data teams to test different datasets and model versions without changing the pipeline code, saving time and reducing errors.

💼 Career

Understanding how to pass parameters to pipelines is essential for MLOps engineers and data scientists to automate and scale machine learning workflows.

1

Create pipeline parameters dictionary

Create a dictionary called pipeline_params with these exact entries: 'dataset_path': '/data/train.csv' and 'model_version': 'v1.0'

MLOps

# Create the pipeline_params dictionary with dataset_path and model_version
# Your code here

Hint

Use curly braces {} to create a dictionary with the exact keys and values.

2

Add run ID variable

Add a variable called run_id and set it to the integer 101

MLOps

pipeline_params = {'dataset_path': '/data/train.csv', 'model_version': 'v1.0'}
# Add the run_id variable and set it to 101
# Your code here

Hint

Just assign the number 101 to a variable named run_id.

3

Write the pipeline run function

Write a function called run_pipeline that takes two parameters: params and run_id. Inside, print the message: "Running pipeline with dataset: {dataset_path}, model version: {model_version}, run ID: {run_id}" using the values from params and run_id

MLOps

pipeline_params = {'dataset_path': '/data/train.csv', 'model_version': 'v1.0'}
run_id = 101
# Define the run_pipeline function below
# Your code here

Hint

Use an f-string to format the print message with values from the params dictionary and the run_id variable.

4

Call the pipeline run function

Call the function run_pipeline with pipeline_params and run_id as arguments to print the pipeline run message

MLOps

pipeline_params = {'dataset_path': '/data/train.csv', 'model_version': 'v1.0'}
run_id = 101

def run_pipeline(params, run_id):
    print(f"Running pipeline with dataset: {params['dataset_path']}, model version: {params['model_version']}, run ID: {run_id}")

# Call run_pipeline with pipeline_params and run_id
# Your code here

Hint

Call the function with the exact variable names pipeline_params and run_id.

Practice

1. What is the main benefit of using parameterized pipeline runs in MLOps?

easy

A. They generate reports after pipeline completion.

B. They automatically fix errors in the pipeline code.

C. They speed up the pipeline execution by parallel processing.

D. They allow customizing pipeline inputs without changing the pipeline code.

Solution

1. Step 1: Understand pipeline parameterization

   Parameterized runs let you pass different inputs to the same pipeline code, making it flexible.

2. Step 2: Identify the main benefit

   This avoids changing the pipeline code for each run, saving time and reducing errors.

3. Final Answer:

   They allow customizing pipeline inputs without changing the pipeline code. -> Option D

4. Quick Check:

   Parameterization = Customize inputs without code change [OK]

Hint: Remember: parameters change inputs, not code [OK]

Common Mistakes:

• Thinking parameters fix code errors
• Confusing parameterization with parallelism
• Assuming parameters generate reports

2. Which of the following is the correct way to pass parameters when triggering a pipeline run using a CLI command?

easy

A. pipeline run -param learning_rate 0.01

B. pipeline run --param learning_rate=0.01

C. pipeline run --parameters learning_rate:0.01

D. pipeline run --param learning_rate:0.01

Solution

1. Step 1: Review common CLI parameter syntax

   Most CLI tools use double dashes and equal signs to pass key-value parameters, like --param key=value.

2. Step 2: Match the correct syntax

   pipeline run --param learning_rate=0.01 uses --param learning_rate=0.01, which is the standard and correct format.

3. Final Answer:

   pipeline run --param learning_rate=0.01 -> Option B

4. Quick Check:

   CLI param syntax = --param key=value [OK]

Hint: Use --param key=value format for CLI parameters [OK]

Common Mistakes:

• Using colon instead of equal sign
• Missing double dashes before param
• Separating key and value with space

3. Given this pipeline run command:
pipeline run --param batch_size=32 --param epochs=10
What will be the values of batch_size and epochs inside the pipeline?

medium

A. batch_size=32, epochs=10

B. batch_size=32, epochs=default

C. batch_size=10, epochs=32

D. batch_size=default, epochs=10

Solution

1. Step 1: Identify parameter assignments in the command

   The command passes batch_size=32 and epochs=10 explicitly.

2. Step 2: Understand parameter values inside the pipeline

   These values override any defaults, so inside the pipeline batch_size=32 and epochs=10.

3. Final Answer:

   batch_size=32, epochs=10 -> Option A

4. Quick Check:

   Passed params = used values inside pipeline [OK]

Hint: Parameters passed override defaults inside pipeline [OK]

Common Mistakes:

• Swapping parameter values
• Assuming defaults when parameters are passed
• Confusing parameter names

4. You run a pipeline with this command:
pipeline run --param learning_rate=0.01 --param epochs
But the pipeline fails to start. What is the most likely cause?

medium

A. Parameters must be passed in a config file, not CLI.

B. Incorrect parameter name learning_rate.

C. Missing value for the parameter epochs.

D. Pipeline does not support parameters.

Solution

1. Step 1: Analyze the command parameters

   The parameter epochs is passed without a value, which is invalid syntax.

2. Step 2: Understand pipeline parameter requirements

   Each parameter must have a value; missing values cause errors and prevent pipeline start.

3. Final Answer:

   Missing value for the parameter epochs. -> Option C

4. Quick Check:

   All params need values [OK]

Hint: Always provide values for all parameters [OK]

Common Mistakes:

• Passing parameters without values
• Assuming pipeline ignores missing values
• Confusing parameter names

5. You want to run the same pipeline with different datasets without changing the pipeline code. Which approach best uses parameterized pipeline runs to achieve this?

hard

A. Pass the dataset path as a parameter when triggering each pipeline run.

B. Create a separate pipeline for each dataset.

C. Hardcode dataset paths inside the pipeline code.

D. Manually edit the pipeline code before each run.

Solution

1. Step 1: Understand the goal

   You want to reuse the same pipeline code but run it on different datasets.

2. Step 2: Use parameterized runs for dataset paths

   Passing dataset paths as parameters lets you run the pipeline multiple times with different inputs without code changes.

3. Step 3: Evaluate other options

   Creating separate pipelines or editing code manually is inefficient and error-prone.

4. Final Answer:

   Pass the dataset path as a parameter when triggering each pipeline run. -> Option A

5. Quick Check:

   Parameterize inputs for reuse [OK]

Hint: Use parameters to swap datasets, not code edits [OK]

Common Mistakes:

• Creating multiple pipelines unnecessarily
• Hardcoding values inside pipeline code
• Editing code before every run