Model drift detection in ML Python - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is model drift in machine learning?

Model drift happens when a machine learning model's performance gets worse over time because the data it sees changes from what it was trained on.

beginner

Name two common types of model drift.

The two common types are:
1. Concept drift: When the relationship between input and output changes.
2. Data drift: When the input data distribution changes but the relationship stays the same.

beginner

Why is detecting model drift important?

Detecting model drift helps keep models accurate and reliable by alerting us when the model needs retraining or updating.

intermediate

What is a simple method to detect data drift?

A simple method is to compare statistics like mean or distribution of new data with the training data using tests like the Kolmogorov-Smirnov test.

intermediate

How can monitoring model performance metrics help in drift detection?

If metrics like accuracy or error rate suddenly change on new data, it can indicate model drift and that the model may no longer be reliable.

What does concept drift mean?

A Model architecture changes

B Change in the input data distribution only

C Change in the relationship between inputs and outputs

D Model training time increases

Concept drift means the way inputs relate to outputs changes, affecting model predictions.

Which metric change might indicate model drift?

A Sudden drop in accuracy

B Increase in training speed

C More features added

D Longer model code

A sudden drop in accuracy on new data suggests the model is not performing well, indicating drift.

Which test can be used to detect data drift?

A T-test for model weights

B Kolmogorov-Smirnov test

C Gradient descent

D Cross-validation

The Kolmogorov-Smirnov test compares distributions to detect changes in data.

What should you do if model drift is detected?

A Delete the model

B Ignore it

C Change the programming language

D Retrain or update the model

Retraining or updating the model helps it adapt to new data and maintain accuracy.

Data drift affects which part of the model?

A Input data distribution

B Model output labels

C Model hyperparameters

D Training algorithm

Data drift means the input data distribution changes over time.

Explain what model drift is and why it matters in simple terms.

Think about how a model can become less accurate over time.

Describe two ways to detect model drift and what actions to take after detection.

Consider checking model results and comparing new data to old data.

Practice

1. What is the main purpose of model drift detection in machine learning?

easy

A. To check if the model's performance decreases over time

B. To increase the model's training speed

C. To reduce the size of the training dataset

D. To improve the model's accuracy during training

Solution

1. Step 1: Understand model drift concept

   Model drift means the model's predictions become less accurate over time due to changes in data or environment.

2. Step 2: Identify the purpose of detection

   Detecting drift helps know when the model needs retraining or updating to maintain performance.

3. Final Answer:

   To check if the model's performance decreases over time -> Option A

4. Quick Check:

   Model drift detection = performance drop check [OK]

Hint: Model drift means performance drops over time [OK]

Common Mistakes:

• Confusing drift detection with training speed improvement
• Thinking drift reduces dataset size
• Assuming drift improves accuracy during training

2. Which of the following is a correct Python syntax to compare model predictions before and after deployment for drift detection?

easy

A. drift = old_preds + new_preds

B. drift = compare(old_preds, new_preds)

C. drift = old_preds == new_preds

D. drift = np.mean(old_preds != new_preds)

Solution

1. Step 1: Understand drift comparison logic

   Drift detection often measures how much predictions differ, so counting mismatches is useful.

2. Step 2: Check syntax correctness

   drift = np.mean(old_preds != new_preds) uses numpy mean of mismatches (old_preds != new_preds), which correctly calculates difference ratio.

3. Final Answer:

   drift = np.mean(old_preds != new_preds) -> Option D

4. Quick Check:

   Correct syntax for prediction difference = drift = np.mean(old_preds != new_preds) [OK]

Hint: Use mean of mismatches to measure prediction drift [OK]

Common Mistakes:

• Using equality operator without aggregation
• Assuming a function 'compare' exists by default
• Adding predictions instead of comparing

3. Given the following code snippet for drift detection, what will be the output?

import numpy as np
old_preds = np.array([1, 0, 1, 1, 0])
new_preds = np.array([1, 1, 1, 0, 0])
drift_score = np.mean(old_preds != new_preds)
print(round(drift_score, 2))

medium

A. 0.40

B. 0.80

C. 0.60

D. 0.20

Solution

1. Step 1: Calculate mismatches between old_preds and new_preds

   Comparing element-wise: [1==1, 0!=1, 1==1, 1!=0, 0==0] gives mismatches at positions 1 and 3 (2 mismatches).

2. Step 2: Compute mean of mismatches

   Number of mismatches = 2, total elements = 5, so drift_score = 2/5 = 0.4.

3. Final Answer:

   0.40 -> Option A

4. Quick Check:

   Mismatch count / total = 0.4 [OK]

Hint: Count mismatches, divide by total elements [OK]

Common Mistakes:

• Counting matches instead of mismatches
• Rounding incorrectly
• Miscounting array length

4. Identify the error in this drift detection code snippet:

old_data = [0.1, 0.2, 0.3]
new_data = [0.15, 0.25, 0.35]
drift = sum(old_data != new_data) / len(old_data)
print(drift)

medium

A. len() function is missing parentheses

B. Using list comparison with != causes an error

C. sum() cannot be used on lists

D. No error, code runs correctly

Solution

1. Step 1: Understand list comparison behavior

   Using != between two lists compares entire lists, not element-wise, so old_data != new_data returns a single boolean, not element-wise array.

2. Step 2: Identify error cause

   sum() expects iterable of numbers, but gets a single boolean, causing a TypeError or wrong result.

3. Final Answer:

   Using list comparison with != causes an error -> Option B

4. Quick Check:

   List != list is not element-wise [OK]

Hint: Use numpy arrays for element-wise comparison [OK]

Common Mistakes:

• Thinking sum() can't be used on lists
• Forgetting parentheses in len()
• Assuming no error occurs

5. You have a deployed model and want to detect drift by comparing feature distributions before and after deployment. Which approach is best to detect drift in continuous features?

hard

A. Calculate accuracy difference between old and new predictions

B. Count how many predictions changed between old and new data

C. Use Kolmogorov-Smirnov test to compare old and new feature distributions

D. Retrain the model immediately without checking data

Solution

1. Step 1: Understand drift detection for continuous features

   Comparing distributions of continuous features requires statistical tests to detect changes in data patterns.

2. Step 2: Identify appropriate statistical test

   Kolmogorov-Smirnov test is a common method to compare two continuous distributions and detect drift.

3. Final Answer:

   Use Kolmogorov-Smirnov test to compare old and new feature distributions -> Option C

4. Quick Check:

   Statistical test for continuous feature drift = Use Kolmogorov-Smirnov test to compare old and new feature distributions [OK]

Hint: Use KS test for continuous feature distribution drift [OK]

Common Mistakes:

• Using accuracy difference for feature drift
• Counting prediction changes instead of feature changes
• Retraining without drift check