Pipeline versioning and reproducibility in MLOps - Mini Project: Build & Apply

Pipeline versioning and reproducibility

📖 Scenario: You are working as a machine learning engineer. Your team needs to ensure that the data processing pipeline is versioned and reproducible. This means that every time the pipeline runs, it uses the exact same code and configuration to produce the same results. This helps in debugging and auditing the model training process.

🎯 Goal: Build a simple pipeline versioning setup using a dictionary to store pipeline steps and a version number. Then, add a configuration variable for the pipeline version. Finally, implement a function that runs the pipeline steps and prints the version used.

📋 What You'll Learn

Create a dictionary called pipeline_steps with exact keys and values

Add a variable called pipeline_version with the exact value 'v1.0'

Write a function called run_pipeline that prints the pipeline version and iterates over pipeline_steps

Print the output of run_pipeline() to show the pipeline version and steps

💡 Why This Matters

🌍 Real World

Versioning and reproducibility in pipelines help teams track changes and ensure consistent results in machine learning workflows.

💼 Career

Understanding pipeline versioning is essential for MLOps engineers to maintain reliable and auditable machine learning systems.

1

Create the pipeline steps dictionary

Create a dictionary called pipeline_steps with these exact entries: 'extract': 'Extract data from source', 'transform': 'Clean and transform data', 'load': 'Load data into database'.

MLOps

# Create the pipeline_steps dictionary with exact keys and values
# Your code here

Need a hint?

Use curly braces {} to create a dictionary. Each key-value pair should be separated by a colon :.

2

Add the pipeline version variable

Add a variable called pipeline_version and set it to the string 'v1.0'.

MLOps

pipeline_steps = {
    'extract': 'Extract data from source',
    'transform': 'Clean and transform data',
    'load': 'Load data into database'
}
# Add the pipeline_version variable below
# Your code here

Need a hint?

Assign the string 'v1.0' to the variable pipeline_version using the equals sign =.

3

Write the run_pipeline function

Write a function called run_pipeline that prints "Running pipeline version: {pipeline_version}" using an f-string. Then use a for loop with variables step and description to iterate over pipeline_steps.items() and print each step and its description in the format "Step: {step} - {description}".

MLOps

pipeline_steps = {
    'extract': 'Extract data from source',
    'transform': 'Clean and transform data',
    'load': 'Load data into database'
}
pipeline_version = 'v1.0'

# Write the run_pipeline function below
# Your code here

Need a hint?

Define a function with def run_pipeline():. Use an f-string inside print() to show the version. Use for step, description in pipeline_steps.items(): to loop through the dictionary.

4

Run the pipeline and print output

Call the function run_pipeline() to print the pipeline version and steps.

MLOps

pipeline_steps = {
    'extract': 'Extract data from source',
    'transform': 'Clean and transform data',
    'load': 'Load data into database'
}
pipeline_version = 'v1.0'

def run_pipeline():
    print(f"Running pipeline version: {pipeline_version}")
    for step, description in pipeline_steps.items():
        print(f"Step: {step} - {description}")

# Call run_pipeline() below
# Your code here

Need a hint?

Simply call run_pipeline() to execute the function and print the output.