Model drift detection in ML Python - ML Experiment: Train & Evaluate

Experiment - Model drift detection

Problem:You have a classification model trained on old data. Over time, the data changes and the model's predictions become less accurate. This is called model drift. You want to detect when the model drift happens.

Current Metrics:Initial model accuracy on old data: 90%. Accuracy on new data: 70%. No drift detection implemented.

Issue:The model performs well on old data but poorly on new data. There is no system to detect when the model's performance drops due to data changes.

Your Task

Implement a simple model drift detection method that compares the distribution of new incoming data with the training data. The goal is to detect drift before accuracy drops below 80%.

Use only Python and standard libraries like numpy, pandas, sklearn.

Do not retrain the model in this task.

Focus on detecting drift using statistical tests or distance metrics.

Hint 1

Hint 2

Hint 3

Solution

ML Python

import numpy as np
import pandas as pd
from sklearn.datasets import make_classification
from scipy.stats import ks_2samp

n_features = 5

# Generate old training data
X_old, y_old = make_classification(n_samples=1000, n_features=n_features, random_state=42)

# Generate new data with drift (shift in feature distribution)
X_new, y_new = make_classification(n_samples=300, n_features=n_features, random_state=24)
X_new += 0.5  # Shift features to simulate drift

# Function to detect drift using KS test
# Returns True if drift detected

def detect_drift(X_ref, X_test, alpha=0.05):
    drift_flags = []
    for i in range(X_ref.shape[1]):
        stat, p_value = ks_2samp(X_ref[:, i], X_test[:, i])
        drift_flags.append(p_value < alpha)
    # If any feature shows drift, flag drift
    return any(drift_flags)

# Detect drift
is_drift = detect_drift(X_old, X_new)
print(f"Drift detected: {is_drift}")

Added a function to compare feature distributions using the Kolmogorov-Smirnov test.

Simulated new data with shifted features to represent drift.

Implemented a simple rule to flag drift if any feature distribution changes significantly.

Results Interpretation

Before: No drift detection, model accuracy dropped from 90% to 70% on new data.

After: Drift detection flagged data change before accuracy dropped below 80%, allowing timely intervention.

Model drift detection helps identify when input data changes enough to affect model performance. Using simple statistical tests on features can alert us early to retrain or update the model.

Bonus Experiment

Try using the Jensen-Shannon divergence to measure drift instead of the KS test.

💡 Hint

Calculate probability distributions of features using histograms and then compute Jensen-Shannon divergence to quantify differences.

Practice

1. What is the main purpose of model drift detection in machine learning?

easy

A. To check if the model's performance decreases over time

B. To increase the model's training speed

C. To reduce the size of the training dataset

D. To improve the model's accuracy during training

Solution

1. Step 1: Understand model drift concept

   Model drift means the model's predictions become less accurate over time due to changes in data or environment.

2. Step 2: Identify the purpose of detection

   Detecting drift helps know when the model needs retraining or updating to maintain performance.

3. Final Answer:

   To check if the model's performance decreases over time -> Option A

4. Quick Check:

   Model drift detection = performance drop check [OK]

Hint: Model drift means performance drops over time [OK]

Common Mistakes:

• Confusing drift detection with training speed improvement
• Thinking drift reduces dataset size
• Assuming drift improves accuracy during training

2. Which of the following is a correct Python syntax to compare model predictions before and after deployment for drift detection?

easy

A. drift = old_preds + new_preds

B. drift = compare(old_preds, new_preds)

C. drift = old_preds == new_preds

D. drift = np.mean(old_preds != new_preds)

Solution

1. Step 1: Understand drift comparison logic

   Drift detection often measures how much predictions differ, so counting mismatches is useful.

2. Step 2: Check syntax correctness

   drift = np.mean(old_preds != new_preds) uses numpy mean of mismatches (old_preds != new_preds), which correctly calculates difference ratio.

3. Final Answer:

   drift = np.mean(old_preds != new_preds) -> Option D

4. Quick Check:

   Correct syntax for prediction difference = drift = np.mean(old_preds != new_preds) [OK]

Hint: Use mean of mismatches to measure prediction drift [OK]

Common Mistakes:

• Using equality operator without aggregation
• Assuming a function 'compare' exists by default
• Adding predictions instead of comparing

3. Given the following code snippet for drift detection, what will be the output?

import numpy as np
old_preds = np.array([1, 0, 1, 1, 0])
new_preds = np.array([1, 1, 1, 0, 0])
drift_score = np.mean(old_preds != new_preds)
print(round(drift_score, 2))

medium

A. 0.40

B. 0.80

C. 0.60

D. 0.20

Solution

1. Step 1: Calculate mismatches between old_preds and new_preds

   Comparing element-wise: [1==1, 0!=1, 1==1, 1!=0, 0==0] gives mismatches at positions 1 and 3 (2 mismatches).

2. Step 2: Compute mean of mismatches

   Number of mismatches = 2, total elements = 5, so drift_score = 2/5 = 0.4.

3. Final Answer:

   0.40 -> Option A

4. Quick Check:

   Mismatch count / total = 0.4 [OK]

Hint: Count mismatches, divide by total elements [OK]

Common Mistakes:

• Counting matches instead of mismatches
• Rounding incorrectly
• Miscounting array length

4. Identify the error in this drift detection code snippet:

old_data = [0.1, 0.2, 0.3]
new_data = [0.15, 0.25, 0.35]
drift = sum(old_data != new_data) / len(old_data)
print(drift)

medium

A. len() function is missing parentheses

B. Using list comparison with != causes an error

C. sum() cannot be used on lists

D. No error, code runs correctly

Solution

1. Step 1: Understand list comparison behavior

   Using != between two lists compares entire lists, not element-wise, so old_data != new_data returns a single boolean, not element-wise array.

2. Step 2: Identify error cause

   sum() expects iterable of numbers, but gets a single boolean, causing a TypeError or wrong result.

3. Final Answer:

   Using list comparison with != causes an error -> Option B

4. Quick Check:

   List != list is not element-wise [OK]

Hint: Use numpy arrays for element-wise comparison [OK]

Common Mistakes:

• Thinking sum() can't be used on lists
• Forgetting parentheses in len()
• Assuming no error occurs

5. You have a deployed model and want to detect drift by comparing feature distributions before and after deployment. Which approach is best to detect drift in continuous features?

hard

A. Calculate accuracy difference between old and new predictions

B. Count how many predictions changed between old and new data

C. Use Kolmogorov-Smirnov test to compare old and new feature distributions

D. Retrain the model immediately without checking data

Solution

1. Step 1: Understand drift detection for continuous features

   Comparing distributions of continuous features requires statistical tests to detect changes in data patterns.

2. Step 2: Identify appropriate statistical test

   Kolmogorov-Smirnov test is a common method to compare two continuous distributions and detect drift.

3. Final Answer:

   Use Kolmogorov-Smirnov test to compare old and new feature distributions -> Option C

4. Quick Check:

   Statistical test for continuous feature drift = Use Kolmogorov-Smirnov test to compare old and new feature distributions [OK]

Hint: Use KS test for continuous feature distribution drift [OK]

Common Mistakes:

• Using accuracy difference for feature drift
• Counting prediction changes instead of feature changes
• Retraining without drift check