Model drift detection in ML Python - Model Metrics & Evaluation

Metrics & Evaluation - Model drift detection

Which metric matters for Model Drift Detection and WHY

Model drift means the model's predictions get worse over time because data changes. To catch this, we watch metrics like accuracy, precision, recall, and F1 score regularly. If these drop, it signals drift.

Also, statistical tests like Population Stability Index (PSI) or Kullback-Leibler divergence check if input data changed. These help spot drift before metrics drop.

Why? Because catching drift early helps fix the model before it makes bad decisions.

Confusion Matrix Example to Understand Drift

Imagine a spam filter model. Here is its confusion matrix at two times:

    Time 1 (Good model):
      TP=90  FP=10
      FN=5   TN=95

    Time 2 (After drift):
      TP=60  FP=40
      FN=30  TN=70
    

At Time 1, precision = 90/(90+10) = 0.9, recall = 90/(90+5) = 0.947. At Time 2, precision = 60/(60+40) = 0.6, recall = 60/(60+30) = 0.667.

The drop in precision and recall shows the model is confused more often, signaling drift.

Precision vs Recall Tradeoff in Drift Detection

When drift happens, precision and recall often drop together. But depending on the use case, one matters more:

• High precision needed: Email spam filter. We want to avoid marking good emails as spam (false positives). So, precision drop is bad.
• High recall needed: Disease detection. We want to catch all sick patients (few false negatives). So, recall drop is bad.

Watching which metric drops helps decide how serious the drift is and what to fix.

Good vs Bad Metric Values for Model Drift

Good: Metrics stay stable over time. For example, accuracy around 90%, precision and recall above 85%. This means the model still works well.

Bad: Sudden drops like accuracy falling from 90% to 70%, precision or recall below 60%. This means the model is confused and likely drifting.

Also, if data distribution metrics like PSI exceed 0.25, it signals significant drift.

Common Pitfalls in Model Drift Detection

• Ignoring data changes: Only watching accuracy can miss drift if data changes but accuracy stays high by chance.
• Data leakage: If future data leaks into training, drift detection is unreliable.
• Overfitting: A model too tuned to old data may show drift quickly but retraining without new data can worsen performance.
• Delayed detection: Checking metrics too rarely can miss early drift signs.

Self Check: Your model has 98% accuracy but 12% recall on fraud. Is it good?

No, it is not good for fraud detection. The high accuracy is misleading because fraud cases are rare. The very low recall (12%) means the model misses most frauds, which is dangerous. For fraud, high recall is critical to catch as many frauds as possible.

Key Result

Monitoring precision, recall, and data distribution metrics over time is key to detecting model drift early and maintaining model reliability.