Model drift detection in ML Python - Model Metrics & Evaluation

Metrics & Evaluation - Model drift detection

Which metric matters for Model Drift Detection and WHY

Model drift means the model's predictions get worse over time because data changes. To catch this, we watch metrics like accuracy, precision, recall, and F1 score regularly. If these drop, it signals drift.

Also, statistical tests like Population Stability Index (PSI) or Kullback-Leibler divergence check if input data changed. These help spot drift before metrics drop.

Why? Because catching drift early helps fix the model before it makes bad decisions.

Confusion Matrix Example to Understand Drift

Imagine a spam filter model. Here is its confusion matrix at two times:

    Time 1 (Good model):
      TP=90  FP=10
      FN=5   TN=95

    Time 2 (After drift):
      TP=60  FP=40
      FN=30  TN=70
    

At Time 1, precision = 90/(90+10) = 0.9, recall = 90/(90+5) = 0.947. At Time 2, precision = 60/(60+40) = 0.6, recall = 60/(60+30) = 0.667.

The drop in precision and recall shows the model is confused more often, signaling drift.

Precision vs Recall Tradeoff in Drift Detection

When drift happens, precision and recall often drop together. But depending on the use case, one matters more:

• High precision needed: Email spam filter. We want to avoid marking good emails as spam (false positives). So, precision drop is bad.
• High recall needed: Disease detection. We want to catch all sick patients (few false negatives). So, recall drop is bad.

Watching which metric drops helps decide how serious the drift is and what to fix.

Good vs Bad Metric Values for Model Drift

Good: Metrics stay stable over time. For example, accuracy around 90%, precision and recall above 85%. This means the model still works well.

Bad: Sudden drops like accuracy falling from 90% to 70%, precision or recall below 60%. This means the model is confused and likely drifting.

Also, if data distribution metrics like PSI exceed 0.25, it signals significant drift.

Common Pitfalls in Model Drift Detection

• Ignoring data changes: Only watching accuracy can miss drift if data changes but accuracy stays high by chance.
• Data leakage: If future data leaks into training, drift detection is unreliable.
• Overfitting: A model too tuned to old data may show drift quickly but retraining without new data can worsen performance.
• Delayed detection: Checking metrics too rarely can miss early drift signs.

Self Check: Your model has 98% accuracy but 12% recall on fraud. Is it good?

No, it is not good for fraud detection. The high accuracy is misleading because fraud cases are rare. The very low recall (12%) means the model misses most frauds, which is dangerous. For fraud, high recall is critical to catch as many frauds as possible.

Key Result

Monitoring precision, recall, and data distribution metrics over time is key to detecting model drift early and maintaining model reliability.

Practice

1. What is the main purpose of model drift detection in machine learning?

easy

A. To check if the model's performance decreases over time

B. To increase the model's training speed

C. To reduce the size of the training dataset

D. To improve the model's accuracy during training

Solution

1. Step 1: Understand model drift concept

   Model drift means the model's predictions become less accurate over time due to changes in data or environment.

2. Step 2: Identify the purpose of detection

   Detecting drift helps know when the model needs retraining or updating to maintain performance.

3. Final Answer:

   To check if the model's performance decreases over time -> Option A

4. Quick Check:

   Model drift detection = performance drop check [OK]

Hint: Model drift means performance drops over time [OK]

Common Mistakes:

• Confusing drift detection with training speed improvement
• Thinking drift reduces dataset size
• Assuming drift improves accuracy during training

2. Which of the following is a correct Python syntax to compare model predictions before and after deployment for drift detection?

easy

A. drift = old_preds + new_preds

B. drift = compare(old_preds, new_preds)

C. drift = old_preds == new_preds

D. drift = np.mean(old_preds != new_preds)

Solution

1. Step 1: Understand drift comparison logic

   Drift detection often measures how much predictions differ, so counting mismatches is useful.

2. Step 2: Check syntax correctness

   drift = np.mean(old_preds != new_preds) uses numpy mean of mismatches (old_preds != new_preds), which correctly calculates difference ratio.

3. Final Answer:

   drift = np.mean(old_preds != new_preds) -> Option D

4. Quick Check:

   Correct syntax for prediction difference = drift = np.mean(old_preds != new_preds) [OK]

Hint: Use mean of mismatches to measure prediction drift [OK]

Common Mistakes:

• Using equality operator without aggregation
• Assuming a function 'compare' exists by default
• Adding predictions instead of comparing

3. Given the following code snippet for drift detection, what will be the output?

import numpy as np
old_preds = np.array([1, 0, 1, 1, 0])
new_preds = np.array([1, 1, 1, 0, 0])
drift_score = np.mean(old_preds != new_preds)
print(round(drift_score, 2))

medium

A. 0.40

B. 0.80

C. 0.60

D. 0.20

Solution

1. Step 1: Calculate mismatches between old_preds and new_preds

   Comparing element-wise: [1==1, 0!=1, 1==1, 1!=0, 0==0] gives mismatches at positions 1 and 3 (2 mismatches).

2. Step 2: Compute mean of mismatches

   Number of mismatches = 2, total elements = 5, so drift_score = 2/5 = 0.4.

3. Final Answer:

   0.40 -> Option A

4. Quick Check:

   Mismatch count / total = 0.4 [OK]

Hint: Count mismatches, divide by total elements [OK]

Common Mistakes:

• Counting matches instead of mismatches
• Rounding incorrectly
• Miscounting array length

4. Identify the error in this drift detection code snippet:

old_data = [0.1, 0.2, 0.3]
new_data = [0.15, 0.25, 0.35]
drift = sum(old_data != new_data) / len(old_data)
print(drift)

medium

A. len() function is missing parentheses

B. Using list comparison with != causes an error

C. sum() cannot be used on lists

D. No error, code runs correctly

Solution

1. Step 1: Understand list comparison behavior

   Using != between two lists compares entire lists, not element-wise, so old_data != new_data returns a single boolean, not element-wise array.

2. Step 2: Identify error cause

   sum() expects iterable of numbers, but gets a single boolean, causing a TypeError or wrong result.

3. Final Answer:

   Using list comparison with != causes an error -> Option B

4. Quick Check:

   List != list is not element-wise [OK]

Hint: Use numpy arrays for element-wise comparison [OK]

Common Mistakes:

• Thinking sum() can't be used on lists
• Forgetting parentheses in len()
• Assuming no error occurs

5. You have a deployed model and want to detect drift by comparing feature distributions before and after deployment. Which approach is best to detect drift in continuous features?

hard

A. Calculate accuracy difference between old and new predictions

B. Count how many predictions changed between old and new data

C. Use Kolmogorov-Smirnov test to compare old and new feature distributions

D. Retrain the model immediately without checking data

Solution

1. Step 1: Understand drift detection for continuous features

   Comparing distributions of continuous features requires statistical tests to detect changes in data patterns.

2. Step 2: Identify appropriate statistical test

   Kolmogorov-Smirnov test is a common method to compare two continuous distributions and detect drift.

3. Final Answer:

   Use Kolmogorov-Smirnov test to compare old and new feature distributions -> Option C

4. Quick Check:

   Statistical test for continuous feature drift = Use Kolmogorov-Smirnov test to compare old and new feature distributions [OK]

Hint: Use KS test for continuous feature distribution drift [OK]

Common Mistakes:

• Using accuracy difference for feature drift
• Counting prediction changes instead of feature changes
• Retraining without drift check