Pipeline versioning and reproducibility in MLOps - Mini Project: Build & Apply

Pipeline versioning and reproducibility

📖 Scenario: You are working as a machine learning engineer. Your team needs to ensure that the data processing pipeline is versioned and reproducible. This means that every time the pipeline runs, it uses the exact same code and configuration to produce the same results. This helps in debugging and auditing the model training process.

🎯 Goal: Build a simple pipeline versioning setup using a dictionary to store pipeline steps and a version number. Then, add a configuration variable for the pipeline version. Finally, implement a function that runs the pipeline steps and prints the version used.

📋 What You'll Learn

Create a dictionary called pipeline_steps with exact keys and values

Add a variable called pipeline_version with the exact value 'v1.0'

Write a function called run_pipeline that prints the pipeline version and iterates over pipeline_steps

Print the output of run_pipeline() to show the pipeline version and steps

💡 Why This Matters

🌍 Real World

Versioning and reproducibility in pipelines help teams track changes and ensure consistent results in machine learning workflows.

💼 Career

Understanding pipeline versioning is essential for MLOps engineers to maintain reliable and auditable machine learning systems.

1

Create the pipeline steps dictionary

Create a dictionary called pipeline_steps with these exact entries: 'extract': 'Extract data from source', 'transform': 'Clean and transform data', 'load': 'Load data into database'.

MLOps

# Create the pipeline_steps dictionary with exact keys and values
# Your code here

Hint

Use curly braces {} to create a dictionary. Each key-value pair should be separated by a colon :.

2

Add the pipeline version variable

Add a variable called pipeline_version and set it to the string 'v1.0'.

MLOps

pipeline_steps = {
    'extract': 'Extract data from source',
    'transform': 'Clean and transform data',
    'load': 'Load data into database'
}
# Add the pipeline_version variable below
# Your code here

Hint

Assign the string 'v1.0' to the variable pipeline_version using the equals sign =.

3

Write the run_pipeline function

Write a function called run_pipeline that prints "Running pipeline version: {pipeline_version}" using an f-string. Then use a for loop with variables step and description to iterate over pipeline_steps.items() and print each step and its description in the format "Step: {step} - {description}".

MLOps

pipeline_steps = {
    'extract': 'Extract data from source',
    'transform': 'Clean and transform data',
    'load': 'Load data into database'
}
pipeline_version = 'v1.0'

# Write the run_pipeline function below
# Your code here

Hint

Define a function with def run_pipeline():. Use an f-string inside print() to show the version. Use for step, description in pipeline_steps.items(): to loop through the dictionary.

4

Run the pipeline and print output

Call the function run_pipeline() to print the pipeline version and steps.

MLOps

pipeline_steps = {
    'extract': 'Extract data from source',
    'transform': 'Clean and transform data',
    'load': 'Load data into database'
}
pipeline_version = 'v1.0'

def run_pipeline():
    print(f"Running pipeline version: {pipeline_version}")
    for step, description in pipeline_steps.items():
        print(f"Step: {step} - {description}")

# Call run_pipeline() below
# Your code here

Hint

Simply call run_pipeline() to execute the function and print the output.

Practice

1. What is the main purpose of pipeline versioning in MLOps?

easy

A. To increase the size of the dataset used

B. To speed up the training process of machine learning models

C. To track changes in workflows and configurations over time

D. To automatically fix bugs in the code

Solution

1. Step 1: Understand pipeline versioning

   Pipeline versioning means keeping track of changes made to the steps and settings in your workflow.

2. Step 2: Identify the main goal

   This helps teams know what changed and when, making it easier to reproduce or fix issues.

3. Final Answer:

   To track changes in workflows and configurations over time -> Option C

4. Quick Check:

   Pipeline versioning = track changes [OK]

Hint: Versioning means tracking changes over time [OK]

Common Mistakes:

• Confusing versioning with speeding up training
• Thinking versioning fixes bugs automatically
• Believing versioning increases dataset size

2. Which of the following is the correct way to fix a random seed in Python for reproducibility in a pipeline?

easy

A. random.seed(42)

B. random.fix_seed(42)

C. seed.random(42)

D. fix.seed(42)

Solution

1. Step 1: Recall Python random seed syntax

   In Python, the random module uses random.seed(value) to fix the seed.

2. Step 2: Check each option

   Only random.seed(42) matches the correct syntax; others are invalid function calls.

3. Final Answer:

   random.seed(42) -> Option A

4. Quick Check:

   Fix seed in Python = random.seed() [OK]

Hint: Use random.seed(value) to fix seed in Python [OK]

Common Mistakes:

• Using incorrect function names like fix_seed or seed.random
• Confusing method order or syntax
• Missing the random module prefix

3. Given this snippet in a pipeline script:

import random
random.seed(10)
print(random.randint(1, 100))
random.seed(10)
print(random.randint(1, 100))

What will be the output?

medium

A. 67 followed by 67

B. 67 followed by a different number

C. Two different random numbers

D. Error due to repeated seed

Solution

1. Step 1: Understand seed effect on random numbers

   Setting the seed to the same value resets the random number generator to the same state.

2. Step 2: Analyze the code output

   Both calls to random.randint(1, 100) after setting seed(10) will produce the same number, 67.

3. Final Answer:

   67 followed by 67 -> Option A

4. Quick Check:

   Same seed = same random output [OK]

Hint: Same seed resets random sequence, repeat outputs [OK]

Common Mistakes:

• Assuming different outputs after resetting seed
• Thinking repeated seed causes error
• Ignoring seed effect on randomness

4. You run a pipeline but get different results each time, even though you fixed the random seed. What is the most likely cause?

medium

A. The random seed was set correctly

B. The pipeline uses non-deterministic operations or external data changes

C. The pipeline versioning is enabled

D. The code has syntax errors

Solution

1. Step 1: Understand reproducibility factors

   Fixing the random seed controls randomness but does not cover external changes or non-deterministic steps.

2. Step 2: Identify cause of varying results

   If results differ despite fixed seed, likely external data or operations like parallelism cause variation.

3. Final Answer:

   The pipeline uses non-deterministic operations or external data changes -> Option B

4. Quick Check:

   Non-determinism breaks reproducibility [OK]

Hint: Check external data and non-deterministic steps [OK]

Common Mistakes:

• Assuming seed fixes all randomness
• Confusing versioning with reproducibility
• Blaming syntax errors for result changes

5. You want to ensure your ML pipeline is fully reproducible across different machines. Which combination is best to achieve this?

hard

A. Only fix random seeds and ignore environment differences

B. Run pipeline without versioning but log outputs manually

C. Use different random seeds each run and update pipeline versions

D. Fix random seeds, use containerized environments, and version pipeline code

Solution

1. Step 1: Identify reproducibility requirements

   Reproducibility needs fixed seeds, consistent environments, and tracking code changes.

2. Step 2: Evaluate options for best practice

   Fix random seeds, use containerized environments, and version pipeline code combines fixing seeds, containerization for environment consistency, and versioning for tracking changes.

3. Final Answer:

   Fix random seeds, use containerized environments, and version pipeline code -> Option D

4. Quick Check:

   Seeds + containers + versioning = reproducibility [OK]

Hint: Combine seeds, containers, and versioning for full reproducibility [OK]

Common Mistakes:

• Ignoring environment differences
• Changing seeds each run
• Skipping pipeline versioning