Model versioning in ML Python - Model Metrics & Evaluation

Metrics & Evaluation - Model versioning

Which metric matters for Model Versioning and WHY

Model versioning helps track different versions of a machine learning model over time. The key metrics to watch are those that show how well each version performs, such as accuracy, precision, recall, and F1 score. These metrics tell us if a new model version is better or worse than the old one. Without tracking these, we can't know if changes improved the model or made it worse.

Confusion Matrix Example for Comparing Versions

Version 1 Confusion Matrix:
          Predicted
          P     N
Actual P  80    20
       N  15    85

Version 2 Confusion Matrix:
          Predicted
          P     N
Actual P  90    10
       N  20    80

Total samples = 200 for each version
    

This matrix helps calculate precision, recall, and accuracy for each version to compare performance.

Precision vs Recall Tradeoff in Model Versioning

When comparing model versions, sometimes one version has higher precision but lower recall, and another has the opposite. For example, a spam filter model version might catch more spam (higher recall) but also mark more good emails as spam (lower precision). Deciding which version to keep depends on what matters more: avoiding false alarms or catching all spam.

Good vs Bad Metric Values in Model Versioning

A good model version shows improved or stable metrics compared to the previous one. For example, if accuracy rises from 85% to 90%, and precision and recall also improve or stay balanced, that is good. A bad version might have lower accuracy or a big drop in recall or precision, meaning it misses more true cases or makes more mistakes.

Common Pitfalls in Model Versioning Metrics

• Accuracy Paradox: A model might have high accuracy but poor recall on important classes, misleading version comparison.
• Data Leakage: If test data leaks into training, metrics look better but don't reflect real performance.
• Overfitting: A new version might perform well on training data but worse on new data, so metrics can be misleading without proper validation.
• Ignoring Business Context: Metrics alone don't tell the full story; consider what errors cost most in your application.

Self-Check Question

Your new model version has 98% accuracy but only 12% recall on fraud cases. Is it good for production? Why or why not?

Answer: No, it is not good. Even though accuracy is high, the model misses most fraud cases (low recall). For fraud detection, catching fraud (high recall) is critical, so this version would fail in real use.

Key Result

Model versioning focuses on comparing key metrics like accuracy, precision, and recall to ensure new versions improve or maintain performance.

Practice

1. What is the main purpose of model versioning in machine learning?

easy

A. To speed up the training process

B. To save different copies of a model as it improves

C. To reduce the size of the dataset

D. To increase the number of features automatically

Solution

1. Step 1: Understand the concept of model versioning

   Model versioning means saving multiple copies of a model at different stages to track improvements.

2. Step 2: Identify the main goal

   The main goal is to keep versions so you can compare, update, or reproduce models easily.

3. Final Answer:

   To save different copies of a model as it improves -> Option B

4. Quick Check:

   Model versioning = saving model copies [OK]

Hint: Think: Why save multiple models? To track changes [OK]

Common Mistakes:

• Confusing versioning with speeding training
• Thinking it reduces dataset size
• Assuming it adds features automatically

2. Which of the following is the correct way to save a model version in Python using joblib?

easy

A. joblib.store(model, 'model_v1.pkl')

B. joblib.save(model, 'model_v1.pkl')

C. joblib.dump(model, 'model_v1.pkl')

D. joblib.write(model, 'model_v1.pkl')

Solution

1. Step 1: Recall joblib syntax for saving models

   The correct function to save a model with joblib is dump.

2. Step 2: Match the correct function and filename

   Using joblib.dump(model, 'model_v1.pkl') saves the model correctly.

3. Final Answer:

   joblib.dump(model, 'model_v1.pkl') -> Option C

4. Quick Check:

   joblib save function = dump [OK]

Hint: Remember: joblib uses dump to save, load to load [OK]

Common Mistakes:

• Using save instead of dump
• Using store or write which don't exist
• Confusing load and dump functions

3. Consider this Python code snippet for saving model versions:

import joblib

model_v1 = {'accuracy': 0.8}
model_v2 = {'accuracy': 0.85}

joblib.dump(model_v1, 'model_v1.pkl')
joblib.dump(model_v2, 'model_v2.pkl')

loaded_model = joblib.load('model_v2.pkl')
print(loaded_model['accuracy'])

What will be the printed output?

medium

A. 0.8

B. model_v2.pkl

C. Error: file not found

D. 0.85

Solution

1. Step 1: Understand the saving and loading process

   Two models are saved with different filenames: 'model_v1.pkl' and 'model_v2.pkl'.

2. Step 2: Identify which model is loaded and printed

   The code loads 'model_v2.pkl' and prints its accuracy value, which is 0.85.

3. Final Answer:

   0.85 -> Option D

4. Quick Check:

   Loaded model accuracy = 0.85 [OK]

Hint: Check which file is loaded before printing [OK]

Common Mistakes:

• Printing accuracy of model_v1 instead of model_v2
• Confusing filename with accuracy value
• Assuming loading causes error

4. You have this code to save model versions but it throws an error:

import joblib

model = {'accuracy': 0.9}
joblib.dump(model, model_v1.pkl)

What is the error and how to fix it?

medium

A. model_v1.pkl is not quoted as a string; add quotes

B. joblib.dump requires three arguments; add one more

C. model must be a trained model object, not a dict

D. joblib module is not imported correctly

Solution

1. Step 1: Identify the syntax error in the filename

   The filename model_v1.pkl is used without quotes, so Python treats it as a variable, causing a NameError.

2. Step 2: Fix by adding quotes around the filename

   Change to joblib.dump(model, 'model_v1.pkl') to pass a string filename.

3. Final Answer:

   model_v1.pkl is not quoted as a string; add quotes -> Option A

4. Quick Check:

   Filename must be a string [OK]

Hint: Always quote filenames as strings in save/load functions [OK]

Common Mistakes:

• Forgetting quotes around filename
• Adding unnecessary extra arguments
• Thinking model type causes error here

5. You want to keep track of multiple model versions with their training accuracy and date. Which approach below best supports clear model versioning and easy retrieval?

hard

A. Save models with filenames like 'model_2023-06-01_acc0.85.pkl' and keep a CSV log with version info

B. Overwrite the same file 'model.pkl' every time you train a new model

C. Save models without any naming pattern and search manually later

D. Save only the best model and delete all others

Solution

1. Step 1: Understand the need for clear versioning

   Clear filenames with date and accuracy help identify model versions easily.

2. Step 2: Use a CSV log to track versions and metadata

   Keeping a CSV log allows easy retrieval and comparison of models.

3. Final Answer:

   Save models with filenames like 'model_2023-06-01_acc0.85.pkl' and keep a CSV log with version info -> Option A

4. Quick Check:

   Clear naming + log = best versioning [OK]

Hint: Use descriptive filenames plus a log file for tracking [OK]

Common Mistakes:

• Overwriting files losing old versions
• Saving without naming pattern causing confusion
• Deleting older models losing history