Model versioning in TensorFlow - Model Metrics & Evaluation

Metrics & Evaluation - Model versioning

Which metric matters for Model Versioning and WHY

When managing different versions of a machine learning model, the key metrics to track are performance metrics like accuracy, precision, recall, and loss. These metrics show if a new model version is better or worse than the previous one. Tracking these helps decide which version to use in real life.

Also, model size and inference speed matter because smaller and faster models are easier to use in real-world apps. So, versioning helps compare these metrics side by side.

Confusion Matrix Example for Model Versions

Suppose we have two model versions for a binary classification task. Here is their confusion matrix:

Version 1 Confusion Matrix:
TP=80, FP=20
FN=15, TN=85

Version 2 Confusion Matrix:
TP=85, FP=15
FN=10, TN=90

Total samples = 200 for each
    

We can calculate precision and recall for each version to see which is better.

Precision vs Recall Tradeoff in Model Versioning

When comparing model versions, one might have higher precision but lower recall, and another the opposite. For example:

• Version 1: Precision = 80 / (80 + 20) = 0.80, Recall = 80 / (80 + 15) ≈ 0.84
• Version 2: Precision = 85 / (85 + 15) = 0.85, Recall = 85 / (85 + 10) ≈ 0.89

Choosing which version to deploy depends on the problem. If missing positives is costly, prefer higher recall. If false alarms are costly, prefer higher precision.

Good vs Bad Metric Values for Model Versioning

A good model version shows improved or stable metrics compared to previous versions. For example:

• Accuracy above 85% for your task
• Precision and recall balanced and above 80%
• Lower loss values
• Smaller model size or faster inference time without losing accuracy

A bad version might have:

• Lower accuracy or recall than before
• Higher false positives or false negatives
• Much larger size or slower speed without metric gains

Common Pitfalls in Model Versioning Metrics

• Ignoring context: A higher accuracy might not mean better if the data changed or the metric is not suited.
• Data leakage: If test data leaks into training, metrics look too good and mislead version choice.
• Overfitting: New version might perform well on training but worse on real data.
• Not tracking inference speed or size: A model might be accurate but too slow or big for deployment.
• Comparing metrics on different datasets: Always compare versions on the same test data.

Self Check: Is a Model with 98% Accuracy but 12% Recall on Fraud Good?

No, this model is not good for fraud detection. Even though accuracy is high, recall is very low. This means the model misses most fraud cases, which is dangerous. For fraud, catching as many frauds as possible (high recall) is critical, even if some false alarms happen.

Key Result

Model versioning focuses on tracking key metrics like accuracy, precision, recall, and model size to choose the best version for deployment.