Parameterized pipeline runs in MLOps - Step-by-Step Execution

Process Flow - Parameterized pipeline runs

Define pipeline with parameters

↓

Trigger pipeline run

↓

Pass parameter values

↓

Pipeline reads parameters

↓

Execute pipeline steps using parameters

↓

Pipeline completes with outputs based on parameters

This flow shows how a pipeline is defined with parameters, triggered with specific values, and runs using those values.

Execution Sample

MLOps

pipeline:
  parameters:
    - name: learning_rate
      type: float
  steps:
    - train_model:
        run: train.py
        args: --lr ${{ parameters.learning_rate }}

A pipeline definition with a learning_rate parameter passed to a training step.

Process Table

Step	Action	Parameter	Value Passed	Effect
1	Pipeline defined	learning_rate	float type, no value yet	Pipeline ready to accept parameter
2	Pipeline triggered	learning_rate	0.01	Parameter value set for this run
3	Step train_model starts	learning_rate	0.01	Training script receives --lr 0.01
4	Training executes	learning_rate	0.01	Model trains with learning rate 0.01
5	Pipeline completes	learning_rate	0.01	Outputs reflect training with 0.01
6	Pipeline triggered again	learning_rate	0.1	New run with different parameter
7	Step train_model starts	learning_rate	0.1	Training script receives --lr 0.1
8	Training executes	learning_rate	0.1	Model trains with learning rate 0.1
9	Pipeline completes	learning_rate	0.1	Outputs reflect training with 0.1

💡 Pipeline runs stop after all steps complete using passed parameters.

Status Tracker

Variable	Start	After Run 1	After Run 2	Final
learning_rate	undefined	0.01	0.1	0.1

Key Moments - 3 Insights

Why does the pipeline need parameters instead of hardcoded values?

Parameters let you run the same pipeline with different inputs without changing the code, as shown in rows 2 and 6 where different values are passed.

What happens if you trigger the pipeline without passing a parameter value?

The pipeline may use a default or fail; in this example, the parameter must be passed as shown in the execution table rows where values are explicitly set.

How does the training step know which parameter value to use?

The pipeline passes the parameter value to the step as an argument, seen in rows 3 and 7 where the training script receives the --lr argument.

Visual Quiz - 3 Questions

Look at the execution table, what is the learning_rate value passed at step 7?

A 0.1

B 0.01

C undefined

D 0.001

💡 Hint

Check the 'Value Passed' column at step 7 in the execution table.

At which step does the pipeline first complete using the parameter value?

A Step 6

B Step 4

C Step 5

D Step 9

💡 Hint

Look for the 'Pipeline completes' action in the execution table.

If you change the parameter learning_rate to 0.05 in the second run, which step would change?

A Step 2 and 3

B Step 7 and 8

C Step 1 only

D Step 9 only

💡 Hint

Parameter values are passed and used in the training step as shown in steps 7 and 8.

Concept Snapshot

Parameterized Pipeline Runs:
- Define parameters in pipeline YAML
- Trigger runs passing parameter values
- Pipeline steps use parameters as inputs
- Enables flexible runs without code changes
- Outputs depend on parameter values

Full Transcript

This visual execution shows how parameterized pipeline runs work in MLOps. First, a pipeline is defined with parameters like learning_rate. When triggered, the pipeline run receives specific values for these parameters. Each step in the pipeline reads the parameter values and uses them, for example passing --lr 0.01 to a training script. The pipeline completes and outputs results based on the parameter. Running the pipeline again with a different parameter value (like 0.1) repeats the process with new inputs. This allows flexible experimentation without changing pipeline code.

Practice

1. What is the main benefit of using parameterized pipeline runs in MLOps?

easy

A. They generate reports after pipeline completion.

B. They automatically fix errors in the pipeline code.

C. They speed up the pipeline execution by parallel processing.

D. They allow customizing pipeline inputs without changing the pipeline code.

Solution

1. Step 1: Understand pipeline parameterization

   Parameterized runs let you pass different inputs to the same pipeline code, making it flexible.

2. Step 2: Identify the main benefit

   This avoids changing the pipeline code for each run, saving time and reducing errors.

3. Final Answer:

   They allow customizing pipeline inputs without changing the pipeline code. -> Option D

4. Quick Check:

   Parameterization = Customize inputs without code change [OK]

Hint: Remember: parameters change inputs, not code [OK]

Common Mistakes:

• Thinking parameters fix code errors
• Confusing parameterization with parallelism
• Assuming parameters generate reports

2. Which of the following is the correct way to pass parameters when triggering a pipeline run using a CLI command?

easy

A. pipeline run -param learning_rate 0.01

B. pipeline run --param learning_rate=0.01

C. pipeline run --parameters learning_rate:0.01

D. pipeline run --param learning_rate:0.01

Solution

1. Step 1: Review common CLI parameter syntax

   Most CLI tools use double dashes and equal signs to pass key-value parameters, like --param key=value.

2. Step 2: Match the correct syntax

   pipeline run --param learning_rate=0.01 uses --param learning_rate=0.01, which is the standard and correct format.

3. Final Answer:

   pipeline run --param learning_rate=0.01 -> Option B

4. Quick Check:

   CLI param syntax = --param key=value [OK]

Hint: Use --param key=value format for CLI parameters [OK]

Common Mistakes:

• Using colon instead of equal sign
• Missing double dashes before param
• Separating key and value with space

3. Given this pipeline run command:
pipeline run --param batch_size=32 --param epochs=10
What will be the values of batch_size and epochs inside the pipeline?

medium

A. batch_size=32, epochs=10

B. batch_size=32, epochs=default

C. batch_size=10, epochs=32

D. batch_size=default, epochs=10

Solution

1. Step 1: Identify parameter assignments in the command

   The command passes batch_size=32 and epochs=10 explicitly.

2. Step 2: Understand parameter values inside the pipeline

   These values override any defaults, so inside the pipeline batch_size=32 and epochs=10.

3. Final Answer:

   batch_size=32, epochs=10 -> Option A

4. Quick Check:

   Passed params = used values inside pipeline [OK]

Hint: Parameters passed override defaults inside pipeline [OK]

Common Mistakes:

• Swapping parameter values
• Assuming defaults when parameters are passed
• Confusing parameter names

4. You run a pipeline with this command:
pipeline run --param learning_rate=0.01 --param epochs
But the pipeline fails to start. What is the most likely cause?

medium

A. Parameters must be passed in a config file, not CLI.

B. Incorrect parameter name learning_rate.

C. Missing value for the parameter epochs.

D. Pipeline does not support parameters.

Solution

1. Step 1: Analyze the command parameters

   The parameter epochs is passed without a value, which is invalid syntax.

2. Step 2: Understand pipeline parameter requirements

   Each parameter must have a value; missing values cause errors and prevent pipeline start.

3. Final Answer:

   Missing value for the parameter epochs. -> Option C

4. Quick Check:

   All params need values [OK]

Hint: Always provide values for all parameters [OK]

Common Mistakes:

• Passing parameters without values
• Assuming pipeline ignores missing values
• Confusing parameter names

5. You want to run the same pipeline with different datasets without changing the pipeline code. Which approach best uses parameterized pipeline runs to achieve this?

hard

A. Pass the dataset path as a parameter when triggering each pipeline run.

B. Create a separate pipeline for each dataset.

C. Hardcode dataset paths inside the pipeline code.

D. Manually edit the pipeline code before each run.

Solution

1. Step 1: Understand the goal

   You want to reuse the same pipeline code but run it on different datasets.

2. Step 2: Use parameterized runs for dataset paths

   Passing dataset paths as parameters lets you run the pipeline multiple times with different inputs without code changes.

3. Step 3: Evaluate other options

   Creating separate pipelines or editing code manually is inefficient and error-prone.

4. Final Answer:

   Pass the dataset path as a parameter when triggering each pipeline run. -> Option A

5. Quick Check:

   Parameterize inputs for reuse [OK]

Hint: Use parameters to swap datasets, not code edits [OK]

Common Mistakes:

• Creating multiple pipelines unnecessarily
• Hardcoding values inside pipeline code
• Editing code before every run