Reproducible training pipelines in MLOps - Step-by-Step Execution

Process Flow - Reproducible training pipelines

Start: Define pipeline steps

↓

Set fixed data version

↓

Set fixed code version

↓

Configure environment (dependencies)

↓

Run training step

↓

Save model and logs

↓

Validate outputs

↓

End: Pipeline reproducible

The pipeline runs step-by-step with fixed data, code, and environment versions to ensure the same results every time.

Execution Sample

MLOps

steps:
  - name: preprocess
    data_version: v1.0
  - name: train
    code_version: abc123
    env: python3.12
  - name: validate
    model_path: ./model.pkl

Defines a simple pipeline with fixed data, code, and environment versions for reproducible training.

Process Table

Step	Action	Version/Config	Result	Notes
1	Load data	data_version=v1.0	Data loaded successfully	Fixed data version ensures same input
2	Setup environment	python3.12 + deps	Environment ready	Consistent environment for all runs
3	Run training	code_version=abc123	Model trained	Code version fixed for reproducibility
4	Save model	model.pkl	Model saved	Output stored for later use
5	Validate model	model.pkl	Validation passed	Checks confirm reproducible output
6	End	-	Pipeline complete	All steps executed with fixed versions

💡 Pipeline stops after all steps complete with fixed versions ensuring reproducibility

Status Tracker

Variable	Start	After Step 1	After Step 3	After Step 5	Final
data_version	unset	v1.0	v1.0	v1.0	v1.0
code_version	unset	unset	abc123	abc123	abc123
environment	unset	unset	python3.12 + deps	python3.12 + deps	python3.12 + deps
model	none	none	trained model object	trained model object	trained model object
validation_status	none	none	none	passed	passed

Key Moments - 3 Insights

Why do we fix data_version and code_version in the pipeline?

Fixing data_version and code_version ensures the pipeline uses the exact same inputs and code every time, which is shown in execution_table rows 1 and 3 where these versions are set and used.

What happens if the environment is not consistent?

If the environment changes, results may differ even with same data and code. Execution_table row 2 shows environment setup which must be consistent to avoid this.

How do we know the pipeline output is reproducible?

Validation step (row 5) confirms the model and results match expected outputs, proving reproducibility.

Visual Quiz - 3 Questions

Look at the execution_table, what is the data_version used at Step 1?

A latest

B v2.0

C v1.0

D abc123

💡 Hint

Check the 'Version/Config' column for Step 1 in execution_table

At which step is the model saved in the pipeline?

A Step 4

B Step 3

C Step 5

D Step 2

💡 Hint

Look for the 'Save model' action in the 'Action' column of execution_table

If the code_version changes, which step's result would most likely be affected?

A Step 2: Setup environment

B Step 3: Run training

C Step 1: Load data

D Step 5: Validate model

💡 Hint

Refer to execution_table row 3 where code_version is used during training

Concept Snapshot

Reproducible training pipelines fix data, code, and environment versions.
Each step runs with these fixed inputs.
Outputs like models are saved and validated.
This ensures same results every run.
Use version control and environment management.

Full Transcript

A reproducible training pipeline runs a series of steps with fixed versions of data, code, and environment. First, it loads data from a specific version to ensure input consistency. Then, it sets up a controlled environment with exact dependencies. Next, it runs training using a fixed code version to guarantee the same logic. The trained model is saved, and validation checks confirm the output matches expectations. This step-by-step process ensures that running the pipeline multiple times produces the same results, which is essential for reliable machine learning workflows.

Practice

1. What is the main goal of a reproducible training pipeline in MLOps?

easy

A. To ensure the training process produces the same results every time

B. To speed up the training by skipping steps

C. To use different data each time for variety

D. To manually adjust parameters during training

Solution

1. Step 1: Understand reproducibility meaning

   Reproducibility means getting the same output when running the same process multiple times.

2. Step 2: Apply to training pipelines

   In training pipelines, reproducibility ensures consistent model results every run.

3. Final Answer:

   To ensure the training process produces the same results every time -> Option A

4. Quick Check:

   Reproducibility = Same results every time [OK]

Hint: Reproducible means repeatable with same results [OK]

Common Mistakes:

• Thinking reproducible means faster training
• Assuming data changes each run
• Believing manual tweaks improve reproducibility

2. Which of the following is the correct way to specify a fixed random seed in a Python training script for reproducibility?

easy

A. seed.random(42)

B. random.set_seed(42)

C. random.seed(42)

D. set.seed(42)

Solution

1. Step 1: Recall Python random module syntax

   Python's random module uses random.seed(value) to fix the seed.

2. Step 2: Check each option

   Only random.seed(42) matches correct Python syntax.

3. Final Answer:

   random.seed(42) -> Option C

4. Quick Check:

   Python random seed = random.seed() [OK]

Hint: Python random seed uses random.seed(value) [OK]

Common Mistakes:

• Using incorrect function names like set_seed
• Swapping argument order
• Confusing with other languages' syntax

3. Given this snippet in a training pipeline script:

import random
random.seed(123)
print(random.randint(1, 10))
random.seed(123)
print(random.randint(1, 10))

What will be the output?

medium

A. Two different random numbers between 1 and 10

B. The same number printed twice

C. An error because seed is set twice

D. Two zeros printed

Solution

1. Step 1: Understand random.seed effect

   Setting random.seed(123) resets the random number generator to a fixed state.

2. Step 2: Analyze the two prints

   Both calls to random.randint(1, 10) after resetting seed produce the same number.

3. Final Answer:

   The same number printed twice -> Option B

4. Quick Check:

   Reset seed = repeat random number [OK]

Hint: Resetting seed repeats random numbers [OK]

Common Mistakes:

• Assuming different numbers after resetting seed
• Expecting error from multiple seed calls
• Thinking zeros are default output

4. You have a training pipeline that uses a Docker container but results differ each run. Which fix will help make it reproducible?

medium

A. Add a fixed random seed in the training code

B. Remove Docker and run on host directly

C. Use different data each time to test robustness

D. Increase batch size to speed training

Solution

1. Step 1: Identify cause of non-reproducibility

   Randomness in training causes different results unless fixed.

2. Step 2: Apply fixed random seed

   Adding a fixed seed ensures same random choices each run, making results reproducible.

3. Final Answer:

   Add a fixed random seed in the training code -> Option A

4. Quick Check:

   Fixed seed fixes randomness [OK]

Hint: Fix randomness with a seed, not by removing Docker [OK]

Common Mistakes:

• Thinking Docker causes randomness
• Changing data to fix reproducibility
• Adjusting batch size unrelated to reproducibility

5. In a complex training pipeline, which combination ensures reproducibility across different machines?

• 1. Fixed random seeds in code
• 2. Containerized environment with exact dependencies
• 3. Using latest library versions without version control
• 4. Logging all hyperparameters and data versions

Choose the best combination.

hard

A. 2 and 3 only

B. 1 and 3 only

C. All four steps

D. 1, 2, and 4 only

Solution

1. Step 1: Evaluate each step's impact

   Fixed seeds, containerized environments, and logging parameters help reproducibility.

2. Step 2: Identify problematic step

   Using latest libraries without version control can cause differences across machines.

3. Final Answer:

   1, 2, and 4 only -> Option D

4. Quick Check:

   Exclude uncontrolled library versions for reproducibility [OK]

Hint: Control seeds, environment, and logs; avoid uncontrolled versions [OK]

Common Mistakes:

• Including latest libraries without version control
• Ignoring environment differences
• Skipping hyperparameter logging