Model drift detection in ML Python

Introduction

Model drift detection helps us know when a machine learning model stops working well because things have changed. It keeps the model useful over time.

When a model is used in real life and data changes over time, like weather or sales data.

When you want to keep a recommendation system accurate as user preferences change.

When monitoring fraud detection models because fraud patterns evolve.

When deploying models in production to catch performance drops early.

When retraining models regularly to decide if retraining is needed.

Syntax

ML Python

from sklearn.metrics import accuracy_score

# Compare old and new data predictions
old_accuracy = accuracy_score(y_true_old, y_pred_old)
new_accuracy = accuracy_score(y_true_new, y_pred_new)

# Check if accuracy dropped significantly
if new_accuracy < old_accuracy - threshold:
    print('Model drift detected')

This example uses accuracy to detect drift by comparing old and new data performance.

You can use other metrics or statistical tests depending on your problem.

Examples

Using F1 score instead of accuracy to detect drift, useful for imbalanced data.

ML Python

from sklearn.metrics import f1_score

old_f1 = f1_score(y_true_old, y_pred_old)
new_f1 = f1_score(y_true_new, y_pred_new)

if new_f1 < old_f1 - 0.05:
    print('Model drift detected')

Using a statistical test (Kolmogorov-Smirnov) to detect if input data changed.

ML Python

from scipy.stats import ks_2samp

# Compare feature distributions
stat, p_value = ks_2samp(feature_old, feature_new)

if p_value < 0.05:
    print('Feature distribution drift detected')

Sample Model

This program trains a simple model, then tests it on original and changed data to detect drift by accuracy drop.

ML Python

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
import numpy as np

# Create initial data
X, y = make_classification(n_samples=1000, n_features=5, random_state=42)

# Split into train and old test data
X_train, X_old_test, y_train, y_old_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Train model
model = LogisticRegression(max_iter=1000)
model.fit(X_train, y_train)

# Predict on old test data
y_pred_old = model.predict(X_old_test)
old_accuracy = accuracy_score(y_old_test, y_pred_old)

# Simulate new data with drift by changing feature distribution
X_new_test = X_old_test + np.random.normal(0.5, 1.0, X_old_test.shape)

# Predict on new test data
y_pred_new = model.predict(X_new_test)
new_accuracy = accuracy_score(y_old_test, y_pred_new)

# Set threshold for drift detection
threshold = 0.05

# Detect drift
if new_accuracy < old_accuracy - threshold:
    print('Model drift detected')
else:
    print('No model drift detected')

# Print accuracies
print(f'Old accuracy: {old_accuracy:.3f}')
print(f'New accuracy: {new_accuracy:.3f}')

Important Notes

Model drift means the model's performance gets worse because data changed.

Detecting drift early helps keep models accurate and trustworthy.

Use simple metrics or statistical tests depending on your data and problem.

Summary

Model drift detection checks if a model's performance drops over time.

It helps decide when to retrain or update the model.

Common methods compare old and new data predictions or feature distributions.

Practice

1. What is the main purpose of model drift detection in machine learning?

easy

A. To check if the model's performance decreases over time

B. To increase the model's training speed

C. To reduce the size of the training dataset

D. To improve the model's accuracy during training

Solution

1. Step 1: Understand model drift concept

   Model drift means the model's predictions become less accurate over time due to changes in data or environment.

2. Step 2: Identify the purpose of detection

   Detecting drift helps know when the model needs retraining or updating to maintain performance.

3. Final Answer:

   To check if the model's performance decreases over time -> Option A

4. Quick Check:

   Model drift detection = performance drop check [OK]

Hint: Model drift means performance drops over time [OK]

Common Mistakes:

• Confusing drift detection with training speed improvement
• Thinking drift reduces dataset size
• Assuming drift improves accuracy during training

2. Which of the following is a correct Python syntax to compare model predictions before and after deployment for drift detection?

easy

A. drift = old_preds + new_preds

B. drift = compare(old_preds, new_preds)

C. drift = old_preds == new_preds

D. drift = np.mean(old_preds != new_preds)

Solution

1. Step 1: Understand drift comparison logic

   Drift detection often measures how much predictions differ, so counting mismatches is useful.

2. Step 2: Check syntax correctness

   drift = np.mean(old_preds != new_preds) uses numpy mean of mismatches (old_preds != new_preds), which correctly calculates difference ratio.

3. Final Answer:

   drift = np.mean(old_preds != new_preds) -> Option D

4. Quick Check:

   Correct syntax for prediction difference = drift = np.mean(old_preds != new_preds) [OK]

Hint: Use mean of mismatches to measure prediction drift [OK]

Common Mistakes:

• Using equality operator without aggregation
• Assuming a function 'compare' exists by default
• Adding predictions instead of comparing

3. Given the following code snippet for drift detection, what will be the output?

import numpy as np
old_preds = np.array([1, 0, 1, 1, 0])
new_preds = np.array([1, 1, 1, 0, 0])
drift_score = np.mean(old_preds != new_preds)
print(round(drift_score, 2))

medium

A. 0.40

B. 0.80

C. 0.60

D. 0.20

Solution

1. Step 1: Calculate mismatches between old_preds and new_preds

   Comparing element-wise: [1==1, 0!=1, 1==1, 1!=0, 0==0] gives mismatches at positions 1 and 3 (2 mismatches).

2. Step 2: Compute mean of mismatches

   Number of mismatches = 2, total elements = 5, so drift_score = 2/5 = 0.4.

3. Final Answer:

   0.40 -> Option A

4. Quick Check:

   Mismatch count / total = 0.4 [OK]

Hint: Count mismatches, divide by total elements [OK]

Common Mistakes:

• Counting matches instead of mismatches
• Rounding incorrectly
• Miscounting array length

4. Identify the error in this drift detection code snippet:

old_data = [0.1, 0.2, 0.3]
new_data = [0.15, 0.25, 0.35]
drift = sum(old_data != new_data) / len(old_data)
print(drift)

medium

A. len() function is missing parentheses

B. Using list comparison with != causes an error

C. sum() cannot be used on lists

D. No error, code runs correctly

Solution

1. Step 1: Understand list comparison behavior

   Using != between two lists compares entire lists, not element-wise, so old_data != new_data returns a single boolean, not element-wise array.

2. Step 2: Identify error cause

   sum() expects iterable of numbers, but gets a single boolean, causing a TypeError or wrong result.

3. Final Answer:

   Using list comparison with != causes an error -> Option B

4. Quick Check:

   List != list is not element-wise [OK]

Hint: Use numpy arrays for element-wise comparison [OK]

Common Mistakes:

• Thinking sum() can't be used on lists
• Forgetting parentheses in len()
• Assuming no error occurs

5. You have a deployed model and want to detect drift by comparing feature distributions before and after deployment. Which approach is best to detect drift in continuous features?

hard

A. Calculate accuracy difference between old and new predictions

B. Count how many predictions changed between old and new data

C. Use Kolmogorov-Smirnov test to compare old and new feature distributions

D. Retrain the model immediately without checking data

Solution

1. Step 1: Understand drift detection for continuous features

   Comparing distributions of continuous features requires statistical tests to detect changes in data patterns.

2. Step 2: Identify appropriate statistical test

   Kolmogorov-Smirnov test is a common method to compare two continuous distributions and detect drift.

3. Final Answer:

   Use Kolmogorov-Smirnov test to compare old and new feature distributions -> Option C

4. Quick Check:

   Statistical test for continuous feature drift = Use Kolmogorov-Smirnov test to compare old and new feature distributions [OK]

Hint: Use KS test for continuous feature distribution drift [OK]

Common Mistakes:

• Using accuracy difference for feature drift
• Counting prediction changes instead of feature changes
• Retraining without drift check