Reproducible training pipelines in MLOps - Step-by-Step Execution

Process Flow - Reproducible training pipelines

Start: Define pipeline steps

↓

Set fixed data version

↓

Set fixed code version

↓

Configure environment (dependencies)

↓

Run training step

↓

Save model and logs

↓

Validate outputs

↓

End: Pipeline reproducible

The pipeline runs step-by-step with fixed data, code, and environment versions to ensure the same results every time.

Execution Sample

MLOps

steps:
  - name: preprocess
    data_version: v1.0
  - name: train
    code_version: abc123
    env: python3.12
  - name: validate
    model_path: ./model.pkl

Defines a simple pipeline with fixed data, code, and environment versions for reproducible training.

Process Table

Step	Action	Version/Config	Result	Notes
1	Load data	data_version=v1.0	Data loaded successfully	Fixed data version ensures same input
2	Setup environment	python3.12 + deps	Environment ready	Consistent environment for all runs
3	Run training	code_version=abc123	Model trained	Code version fixed for reproducibility
4	Save model	model.pkl	Model saved	Output stored for later use
5	Validate model	model.pkl	Validation passed	Checks confirm reproducible output
6	End	-	Pipeline complete	All steps executed with fixed versions

💡 Pipeline stops after all steps complete with fixed versions ensuring reproducibility

Status Tracker

Variable	Start	After Step 1	After Step 3	After Step 5	Final
data_version	unset	v1.0	v1.0	v1.0	v1.0
code_version	unset	unset	abc123	abc123	abc123
environment	unset	unset	python3.12 + deps	python3.12 + deps	python3.12 + deps
model	none	none	trained model object	trained model object	trained model object
validation_status	none	none	none	passed	passed

Key Moments - 3 Insights

Why do we fix data_version and code_version in the pipeline?

Fixing data_version and code_version ensures the pipeline uses the exact same inputs and code every time, which is shown in execution_table rows 1 and 3 where these versions are set and used.

What happens if the environment is not consistent?

If the environment changes, results may differ even with same data and code. Execution_table row 2 shows environment setup which must be consistent to avoid this.

How do we know the pipeline output is reproducible?

Validation step (row 5) confirms the model and results match expected outputs, proving reproducibility.

Visual Quiz - 3 Questions

Look at the execution_table, what is the data_version used at Step 1?

A latest

B v2.0

C v1.0

D abc123

💡 Hint

Check the 'Version/Config' column for Step 1 in execution_table

At which step is the model saved in the pipeline?

A Step 4

B Step 3

C Step 5

D Step 2

💡 Hint

Look for the 'Save model' action in the 'Action' column of execution_table

If the code_version changes, which step's result would most likely be affected?

A Step 2: Setup environment

B Step 3: Run training

C Step 1: Load data

D Step 5: Validate model

💡 Hint

Refer to execution_table row 3 where code_version is used during training

Concept Snapshot

Reproducible training pipelines fix data, code, and environment versions.
Each step runs with these fixed inputs.
Outputs like models are saved and validated.
This ensures same results every run.
Use version control and environment management.

Full Transcript

A reproducible training pipeline runs a series of steps with fixed versions of data, code, and environment. First, it loads data from a specific version to ensure input consistency. Then, it sets up a controlled environment with exact dependencies. Next, it runs training using a fixed code version to guarantee the same logic. The trained model is saved, and validation checks confirm the output matches expectations. This step-by-step process ensures that running the pipeline multiple times produces the same results, which is essential for reliable machine learning workflows.