Monitoring model performance in ML Python - Model Metrics & Evaluation

Metrics & Evaluation - Monitoring model performance

Which metric matters for monitoring model performance and WHY

When we watch how a model works over time, we want to see if it keeps making good predictions. The main metrics to check are accuracy, precision, recall, and F1 score. These tell us if the model is still correct and balanced in its guesses. We also look at loss to see if the model’s errors are growing. Monitoring these helps catch problems early, like if the model starts to forget or if the data changes.

Confusion matrix example for monitoring

      Actual \ Predicted | Positive | Negative
      -------------------|----------|---------
      Positive           |    80    |   20    
      Negative           |    10    |   90    

      Total samples = 200

      Precision = 80 / (80 + 10) = 0.89
      Recall = 80 / (80 + 20) = 0.80
      Accuracy = (80 + 90) / 200 = 0.85
    

This matrix helps us see if the model is confusing positive and negative cases. Watching it over time shows if mistakes increase.

Precision vs Recall tradeoff with examples

Sometimes, improving one metric lowers the other. For example:

• Spam filter: High precision means fewer good emails marked as spam. We want to avoid losing important emails.
• Cancer detection: High recall means catching most cancer cases. Missing a case is very bad.

When monitoring, if precision drops but recall stays high, the model may mark too many good cases as bad. If recall drops, it may miss important cases. We watch these to keep the balance right.

What good vs bad metric values look like when monitoring

Good: Metrics stay stable or improve over time. For example, accuracy around 90%, precision and recall both above 85%, and loss stays low or decreases.

Bad: Metrics suddenly drop or slowly decline. For example, accuracy falls below 70%, precision or recall drop below 50%, or loss increases. This means the model may be confused or data has changed.

Common pitfalls when monitoring model performance

• Accuracy paradox: High accuracy can be misleading if data is unbalanced. For example, 95% accuracy on 95% negative data means the model ignores positives.
• Data leakage: If future data leaks into training, metrics look great but fail in real use.
• Overfitting indicators: Training metrics improve but test or live metrics worsen, showing the model learned noise, not real patterns.
• Ignoring metric trends: Small drops over time can signal problems before big failures.

Self-check question

Your model has 98% accuracy but only 12% recall on fraud cases. Is it good for production? Why or why not?

Answer: No, it is not good. Even though accuracy is high, the model misses 88% of fraud cases (low recall). This means many frauds go undetected, which is risky. For fraud detection, high recall is critical.

Key Result

Monitoring accuracy, precision, recall, and loss over time helps detect when a model's performance degrades or data changes.