Model registry in ML Python - Model Metrics & Evaluation

Metrics & Evaluation - Model registry

Which metric matters for Model Registry and WHY

A model registry is a system to store and manage machine learning models. It tracks versions, metadata, and performance metrics. The key metrics here are model performance metrics like accuracy, precision, recall, and F1 score. These metrics help decide which model version is best to use or deploy.

Why? Because the registry helps compare models fairly and pick the best one. It also stores metadata like training data, parameters, and evaluation results to keep track of model quality over time.

Confusion Matrix or Equivalent Visualization

When models are registered, their confusion matrices are often saved as part of their evaluation. Here is an example confusion matrix for a binary classifier:

      | Predicted Positive | Predicted Negative |
      |--------------------|--------------------|
      | True Positive (TP) = 80  | False Positive (FP) = 10 |
      | False Negative (FN) = 20 | True Negative (TN) = 90  |
    

This matrix helps calculate precision, recall, and accuracy, which are stored in the registry for each model version.

Precision vs Recall Tradeoff with Examples

Models in the registry often have different precision and recall values. Choosing which model to deploy depends on the tradeoff:

• High Precision: Few false alarms. Good for spam filters so real emails are not marked spam.
• High Recall: Few missed cases. Important for cancer detection to catch all patients.

The registry helps compare these tradeoffs by storing both metrics for each model version.

What Good vs Bad Metric Values Look Like

Good model metrics in the registry mean:

• High accuracy (e.g., > 90%) on test data
• Balanced precision and recall (e.g., both > 80%)
• Stable performance across versions

Bad metrics mean:

• Low accuracy (e.g., < 60%)
• Very low recall or precision (e.g., < 50%)
• Big drops in performance compared to previous versions

The registry flags these to avoid deploying poor models.

Common Metrics Pitfalls in Model Registry

• Accuracy Paradox: High accuracy but poor recall on rare classes can mislead model choice.
• Data Leakage: Metrics look great if test data leaks into training, but real-world performance drops.
• Overfitting: Metrics very high on training but low on test data indicate overfitting.
• Ignoring Metric Context: Not considering which metric matters for the problem leads to wrong model selection.

Self Check: Is a Model with 98% Accuracy but 12% Recall on Fraud Good?

No, this model is not good for fraud detection. Even though accuracy is high, recall is very low. This means it misses most fraud cases, which is dangerous. In fraud detection, catching fraud (high recall) is more important than just overall accuracy.

The model registry would flag this model as unsuitable for deployment despite the high accuracy.

Key Result

Model registry tracks key metrics like precision, recall, and accuracy to help pick the best model version for deployment.