Model versioning in ML Python - ML Experiment: Train & Evaluate

Experiment - Model versioning

Problem:You have trained a machine learning model and saved it as a file. Now you want to keep track of different versions of your model as you improve it, so you can compare and use the best one later.

Current Metrics:Model accuracy: 85%, saved as 'model_v1.pkl'

Issue:No system to manage multiple model versions. It is hard to know which model is best or to roll back to a previous version.

Your Task

Create a simple model versioning system that saves models with version numbers and allows loading a specific version. Demonstrate saving two versions and loading them to compare accuracy.

Use Python and scikit-learn for model training.

Save models as files with version numbers in the filename.

Do not use complex version control tools; keep it simple and clear.

Hint 1

Hint 2

Hint 3

Hint 4

Solution

ML Python

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score
import joblib

# Load data
iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(iris.data, iris.target, test_size=0.3, random_state=42)

# Train first model version
model_v1 = RandomForestClassifier(n_estimators=10, random_state=42)
model_v1.fit(X_train, y_train)
joblib.dump(model_v1, 'model_v1.pkl')

# Evaluate first model
preds_v1 = model_v1.predict(X_test)
acc_v1 = accuracy_score(y_test, preds_v1)

# Train second model version with more trees
model_v2 = RandomForestClassifier(n_estimators=50, random_state=42)
model_v2.fit(X_train, y_train)
joblib.dump(model_v2, 'model_v2.pkl')

# Evaluate second model
preds_v2 = model_v2.predict(X_test)
acc_v2 = accuracy_score(y_test, preds_v2)

# Load models from files
loaded_v1 = joblib.load('model_v1.pkl')
loaded_v2 = joblib.load('model_v2.pkl')

# Confirm loaded models accuracy
loaded_acc_v1 = accuracy_score(y_test, loaded_v1.predict(X_test))
loaded_acc_v2 = accuracy_score(y_test, loaded_v2.predict(X_test))

print(f"Model v1 accuracy: {acc_v1:.2f}")
print(f"Model v2 accuracy: {acc_v2:.2f}")
print(f"Loaded model v1 accuracy: {loaded_acc_v1:.2f}")
print(f"Loaded model v2 accuracy: {loaded_acc_v2:.2f}")

Added saving model files with version numbers in filenames.

Trained two models with different hyperparameters to create versions.

Loaded saved models to verify versioning works correctly.

Results Interpretation

Before: Only one model saved as 'model_v1.pkl' with accuracy 85%. No way to track improvements.

After: Two models saved as 'model_v1.pkl' and 'model_v2.pkl' with accuracies 91% and 96%. Can load any version to compare or use.

Model versioning helps keep track of improvements and makes it easy to use or roll back to previous models. Saving models with clear version numbers is a simple and effective approach.

Bonus Experiment

Implement a simple function that lists all saved model versions in the current folder and allows the user to select one to load.

💡 Hint

Use Python's os module to list files matching 'model_v*.pkl' and let the user input the version number to load.

Practice

1. What is the main purpose of model versioning in machine learning?

easy

A. To speed up the training process

B. To save different copies of a model as it improves

C. To reduce the size of the dataset

D. To increase the number of features automatically

Solution

1. Step 1: Understand the concept of model versioning

   Model versioning means saving multiple copies of a model at different stages to track improvements.

2. Step 2: Identify the main goal

   The main goal is to keep versions so you can compare, update, or reproduce models easily.

3. Final Answer:

   To save different copies of a model as it improves -> Option B

4. Quick Check:

   Model versioning = saving model copies [OK]

Hint: Think: Why save multiple models? To track changes [OK]

Common Mistakes:

• Confusing versioning with speeding training
• Thinking it reduces dataset size
• Assuming it adds features automatically

2. Which of the following is the correct way to save a model version in Python using joblib?

easy

A. joblib.store(model, 'model_v1.pkl')

B. joblib.save(model, 'model_v1.pkl')

C. joblib.dump(model, 'model_v1.pkl')

D. joblib.write(model, 'model_v1.pkl')

Solution

1. Step 1: Recall joblib syntax for saving models

   The correct function to save a model with joblib is dump.

2. Step 2: Match the correct function and filename

   Using joblib.dump(model, 'model_v1.pkl') saves the model correctly.

3. Final Answer:

   joblib.dump(model, 'model_v1.pkl') -> Option C

4. Quick Check:

   joblib save function = dump [OK]

Hint: Remember: joblib uses dump to save, load to load [OK]

Common Mistakes:

• Using save instead of dump
• Using store or write which don't exist
• Confusing load and dump functions

3. Consider this Python code snippet for saving model versions:

import joblib

model_v1 = {'accuracy': 0.8}
model_v2 = {'accuracy': 0.85}

joblib.dump(model_v1, 'model_v1.pkl')
joblib.dump(model_v2, 'model_v2.pkl')

loaded_model = joblib.load('model_v2.pkl')
print(loaded_model['accuracy'])

What will be the printed output?

medium

A. 0.8

B. model_v2.pkl

C. Error: file not found

D. 0.85

Solution

1. Step 1: Understand the saving and loading process

   Two models are saved with different filenames: 'model_v1.pkl' and 'model_v2.pkl'.

2. Step 2: Identify which model is loaded and printed

   The code loads 'model_v2.pkl' and prints its accuracy value, which is 0.85.

3. Final Answer:

   0.85 -> Option D

4. Quick Check:

   Loaded model accuracy = 0.85 [OK]

Hint: Check which file is loaded before printing [OK]

Common Mistakes:

• Printing accuracy of model_v1 instead of model_v2
• Confusing filename with accuracy value
• Assuming loading causes error

4. You have this code to save model versions but it throws an error:

import joblib

model = {'accuracy': 0.9}
joblib.dump(model, model_v1.pkl)

What is the error and how to fix it?

medium

A. model_v1.pkl is not quoted as a string; add quotes

B. joblib.dump requires three arguments; add one more

C. model must be a trained model object, not a dict

D. joblib module is not imported correctly

Solution

1. Step 1: Identify the syntax error in the filename

   The filename model_v1.pkl is used without quotes, so Python treats it as a variable, causing a NameError.

2. Step 2: Fix by adding quotes around the filename

   Change to joblib.dump(model, 'model_v1.pkl') to pass a string filename.

3. Final Answer:

   model_v1.pkl is not quoted as a string; add quotes -> Option A

4. Quick Check:

   Filename must be a string [OK]

Hint: Always quote filenames as strings in save/load functions [OK]

Common Mistakes:

• Forgetting quotes around filename
• Adding unnecessary extra arguments
• Thinking model type causes error here

5. You want to keep track of multiple model versions with their training accuracy and date. Which approach below best supports clear model versioning and easy retrieval?

hard

A. Save models with filenames like 'model_2023-06-01_acc0.85.pkl' and keep a CSV log with version info

B. Overwrite the same file 'model.pkl' every time you train a new model

C. Save models without any naming pattern and search manually later

D. Save only the best model and delete all others

Solution

1. Step 1: Understand the need for clear versioning

   Clear filenames with date and accuracy help identify model versions easily.

2. Step 2: Use a CSV log to track versions and metadata

   Keeping a CSV log allows easy retrieval and comparison of models.

3. Final Answer:

   Save models with filenames like 'model_2023-06-01_acc0.85.pkl' and keep a CSV log with version info -> Option A

4. Quick Check:

   Clear naming + log = best versioning [OK]

Hint: Use descriptive filenames plus a log file for tracking [OK]

Common Mistakes:

• Overwriting files losing old versions
• Saving without naming pattern causing confusion
• Deleting older models losing history