Parameterized pipeline runs in MLOps - Commands & Configuration

Introduction

Sometimes you want to run the same pipeline but with different inputs or settings. Parameterized pipeline runs let you do this easily by changing values without rewriting the whole pipeline.

When you want to test your machine learning model with different training data sizes.

When you need to run the same data processing steps but for different dates or time ranges.

When you want to try different hyperparameters for your model without changing the code.

When you want to reuse the same pipeline for different projects by just changing input paths.

When you want to automate running pipelines with different configurations in a CI/CD system.

Commands

This command runs the pipeline script with parameters param1 set to 10 and param2 set to 'test_value'. It shows how to pass parameters to customize the run.

Terminal

python my_pipeline.py --param1 10 --param2 test_value

Expected Output

Starting pipeline with param1=10 and param2='test_value'

Step 1 completed

Step 2 completed

Pipeline finished successfully

→

--param1 - Sets the first parameter value for the pipeline

→

--param2 - Sets the second parameter value for the pipeline

Runs the same pipeline but with different parameter values to show how easy it is to change inputs without code changes.

Terminal

python my_pipeline.py --param1 20 --param2 another_test

Expected Output

Starting pipeline with param1=20 and param2='another_test'

Step 1 completed

Step 2 completed

Pipeline finished successfully

→

--param1 - Changes the first parameter to 20

→

--param2 - Changes the second parameter to 'another_test'

Key Concept

If you remember nothing else from this pattern, remember: parameterized runs let you reuse the same pipeline code with different inputs easily.

Code Example

MLOps

import argparse

def run_pipeline(param1=5, param2='default'):
    print(f"Starting pipeline with param1={param1} and param2='{param2}'")
    # Simulate steps
    print("Step 1 completed")
    print("Step 2 completed")
    print("Pipeline finished successfully")

if __name__ == '__main__':
    parser = argparse.ArgumentParser(description='Run parameterized pipeline')
    parser.add_argument('--param1', type=int, default=5, help='First parameter')
    parser.add_argument('--param2', type=str, default='default', help='Second parameter')
    args = parser.parse_args()
    run_pipeline(param1=args.param1, param2=args.param2)

Common Mistakes

Not defining default values for parameters in the pipeline code.

This causes the pipeline to fail if parameters are not passed every time.

Always set sensible default values for parameters so the pipeline can run without explicit inputs.

Passing parameters with wrong names or typos.

The pipeline will not recognize them and may ignore or error out.

Check parameter names carefully and use consistent naming in code and commands.

Summary

Use command-line parameters to run the same pipeline with different inputs.

Set default values in your pipeline code to avoid errors when parameters are missing.

Changing parameters is easy and does not require changing the pipeline code itself.