Reproducible training pipelines in MLOps - Commands & Configuration

Introduction

Training machine learning models often involves many steps and settings. Reproducible training pipelines help you run these steps the same way every time, so you get consistent results and can track what you did.

When you want to train a model and be sure you can repeat the exact same process later.

When you need to compare different model versions and know exactly what changed.

When you want to share your training process with teammates so they can run it too.

When you want to automate training so it runs without manual steps.

When you want to keep track of data, code, and parameters used in training.

Commands

This command runs the ML training pipeline defined in the current directory using MLflow. It ensures the same steps and parameters are used every time.

Terminal

mlflow run .

Expected Output

2024/06/01 12:00:00 INFO mlflow.projects: === Created directory /tmp/mlruns for run data ===

2024/06/01 12:00:01 INFO mlflow.projects: === Running command 'python train.py --alpha 0.5 --l1_ratio 0.1' ===

Training model with alpha=0.5 and l1_ratio=0.1

Model training complete

2024/06/01 12:00:10 INFO mlflow.projects: === Run (ID 1234567890abcdef) succeeded ===

This command starts the MLflow tracking UI in your browser. You can see all runs, parameters, and results to compare and reproduce training.

Terminal

mlflow ui

Expected Output

2024/06/01 12:01:00 INFO mlflow.server: Starting MLflow UI at http://127.0.0.1:5000

This runs the training pipeline again but with different parameters. It shows how you can reproduce training with changes tracked.

Terminal

mlflow run . -P alpha=0.7 -P l1_ratio=0.2

Expected Output

2024/06/01 12:02:00 INFO mlflow.projects: === Running command 'python train.py --alpha 0.7 --l1_ratio 0.2' ===

Training model with alpha=0.7 and l1_ratio=0.2

Model training complete

2024/06/01 12:02:10 INFO mlflow.projects: === Run (ID abcdef1234567890) succeeded ===

→

-P - Set a parameter value for the run

Key Concept

If you remember nothing else from this pattern, remember: reproducible pipelines let you run, track, and share model training exactly the same way every time.

Code Example

MLOps

import mlflow
import mlflow.sklearn
from sklearn.linear_model import ElasticNet
from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error, r2_score

# Load data
X, y = load_diabetes(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Parameters
alpha = 0.5
l1_ratio = 0.1

with mlflow.start_run():
    model = ElasticNet(alpha=alpha, l1_ratio=l1_ratio, random_state=42)
    model.fit(X_train, y_train)
    predictions = model.predict(X_test)

    mse = mean_squared_error(y_test, predictions)
    r2 = r2_score(y_test, predictions)

    mlflow.log_param("alpha", alpha)
    mlflow.log_param("l1_ratio", l1_ratio)
    mlflow.log_metric("mse", mse)
    mlflow.log_metric("r2", r2)
    mlflow.sklearn.log_model(model, "model")

    print(f"Run complete with mse={mse:.4f} and r2={r2:.4f}")

Common Mistakes

Not specifying parameters explicitly when running the pipeline

This causes runs to use default or old parameters, making results inconsistent and hard to reproduce.

Always pass parameters explicitly using flags like -P to control training settings.

Not using a tracking tool like MLflow to log runs

Without tracking, you lose history of what was run and cannot compare or reproduce results easily.

Use MLflow or similar tools to log parameters, metrics, and artifacts for every run.

Changing code or data without version control

Changes outside the pipeline cause runs to differ and break reproducibility.

Keep code and data under version control and link runs to specific versions.

Summary

Use 'mlflow run .' to execute your training pipeline reproducibly with tracked parameters.

Use 'mlflow ui' to view and compare all training runs and their results.

Pass parameters explicitly with '-P' flags to control training settings and keep runs consistent.

Practice

1. What is the main goal of a reproducible training pipeline in MLOps?

easy

A. To ensure the training process produces the same results every time

B. To speed up the training by skipping steps

C. To use different data each time for variety

D. To manually adjust parameters during training

Solution

1. Step 1: Understand reproducibility meaning

   Reproducibility means getting the same output when running the same process multiple times.

2. Step 2: Apply to training pipelines

   In training pipelines, reproducibility ensures consistent model results every run.

3. Final Answer:

   To ensure the training process produces the same results every time -> Option A

4. Quick Check:

   Reproducibility = Same results every time [OK]

Hint: Reproducible means repeatable with same results [OK]

Common Mistakes:

• Thinking reproducible means faster training
• Assuming data changes each run
• Believing manual tweaks improve reproducibility

2. Which of the following is the correct way to specify a fixed random seed in a Python training script for reproducibility?

easy

A. seed.random(42)

B. random.set_seed(42)

C. random.seed(42)

D. set.seed(42)

Solution

1. Step 1: Recall Python random module syntax

   Python's random module uses random.seed(value) to fix the seed.

2. Step 2: Check each option

   Only random.seed(42) matches correct Python syntax.

3. Final Answer:

   random.seed(42) -> Option C

4. Quick Check:

   Python random seed = random.seed() [OK]

Hint: Python random seed uses random.seed(value) [OK]

Common Mistakes:

• Using incorrect function names like set_seed
• Swapping argument order
• Confusing with other languages' syntax

3. Given this snippet in a training pipeline script:

import random
random.seed(123)
print(random.randint(1, 10))
random.seed(123)
print(random.randint(1, 10))

What will be the output?

medium

A. Two different random numbers between 1 and 10

B. The same number printed twice

C. An error because seed is set twice

D. Two zeros printed

Solution

1. Step 1: Understand random.seed effect

   Setting random.seed(123) resets the random number generator to a fixed state.

2. Step 2: Analyze the two prints

   Both calls to random.randint(1, 10) after resetting seed produce the same number.

3. Final Answer:

   The same number printed twice -> Option B

4. Quick Check:

   Reset seed = repeat random number [OK]

Hint: Resetting seed repeats random numbers [OK]

Common Mistakes:

• Assuming different numbers after resetting seed
• Expecting error from multiple seed calls
• Thinking zeros are default output

4. You have a training pipeline that uses a Docker container but results differ each run. Which fix will help make it reproducible?

medium

A. Add a fixed random seed in the training code

B. Remove Docker and run on host directly

C. Use different data each time to test robustness

D. Increase batch size to speed training

Solution

1. Step 1: Identify cause of non-reproducibility

   Randomness in training causes different results unless fixed.

2. Step 2: Apply fixed random seed

   Adding a fixed seed ensures same random choices each run, making results reproducible.

3. Final Answer:

   Add a fixed random seed in the training code -> Option A

4. Quick Check:

   Fixed seed fixes randomness [OK]

Hint: Fix randomness with a seed, not by removing Docker [OK]

Common Mistakes:

• Thinking Docker causes randomness
• Changing data to fix reproducibility
• Adjusting batch size unrelated to reproducibility

5. In a complex training pipeline, which combination ensures reproducibility across different machines?

• 1. Fixed random seeds in code
• 2. Containerized environment with exact dependencies
• 3. Using latest library versions without version control
• 4. Logging all hyperparameters and data versions

Choose the best combination.

hard

A. 2 and 3 only

B. 1 and 3 only

C. All four steps

D. 1, 2, and 4 only

Solution

1. Step 1: Evaluate each step's impact

   Fixed seeds, containerized environments, and logging parameters help reproducibility.

2. Step 2: Identify problematic step

   Using latest libraries without version control can cause differences across machines.

3. Final Answer:

   1, 2, and 4 only -> Option D

4. Quick Check:

   Exclude uncontrolled library versions for reproducibility [OK]

Hint: Control seeds, environment, and logs; avoid uncontrolled versions [OK]

Common Mistakes:

• Including latest libraries without version control
• Ignoring environment differences
• Skipping hyperparameter logging