Model drift detection in ML Python - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

What is model drift in machine learning?

Which of the following best describes model drift?

A When the data distribution changes causing the model's predictions to become less accurate

B When the model's performance improves over time without retraining

C When the model is trained on more data than originally planned

D When the model's architecture is changed during training

💡 Hint

Think about what happens if the world changes but the model stays the same.

Model drift happens when the data the model sees changes over time, making its predictions less accurate because it was trained on old data.

❓ Metrics

intermediate

Detecting model drift using metrics

You have a classification model deployed in production. Which metric change would most likely indicate model drift?

A Training loss decreases during model training

B Precision and recall remain constant

C Sudden drop in F1-score on new data compared to training data

D Accuracy increases steadily over time

💡 Hint

Model drift usually hurts performance on new data.

A sudden drop in F1-score on new data means the model is not predicting well anymore, which is a sign of drift.

❓ Predict Output

advanced

Output of drift detection code snippet

What is the output of this Python code that compares feature distributions to detect drift?

ML Python

import numpy as np
from scipy.stats import ks_2samp

reference = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9])
new_data = np.array([2, 3, 4, 5, 6, 7, 8, 9, 10])

stat, p_value = ks_2samp(reference, new_data)

if p_value < 0.05:
    print('Drift detected')
else:
    print('No drift detected')

A TypeError

B Drift detected

C SyntaxError

D No drift detected

💡 Hint

The Kolmogorov-Smirnov test checks if two samples come from the same distribution.

The p-value is above 0.05, so the test does not detect a significant difference between distributions.

❓ Hyperparameter

advanced

Choosing parameters for drift detection sensitivity

In a drift detection system using a statistical test, which parameter controls how sensitive the system is to detecting drift?

A The significance level (alpha) threshold for the test

B The number of features in the dataset

C The learning rate of the model

D The batch size of data processed

💡 Hint

This parameter decides how small the p-value must be to say drift exists.

The significance level (alpha) sets the cutoff p-value below which drift is declared, controlling sensitivity.

🔧 Debug

expert

Why does this drift detection code fail to detect drift?

Consider this code snippet for detecting drift using population stability index (PSI). Why does it fail to detect drift when new data distribution changes significantly? ```python import numpy as np def psi(expected, actual, buckets=10): def scale_range(input, min, max): input = input - np.min(input) input = input / np.max(input) * (max - min) input = input + min return input breakpoints = np.arange(0, buckets + 1) / buckets * 100 expected_percents = np.histogram(scale_range(expected, 0, 100), bins=breakpoints)[0] / len(expected) actual_percents = np.histogram(scale_range(actual, 0, 100), bins=breakpoints)[0] / len(actual) psi_value = np.sum((expected_percents - actual_percents) * np.log(expected_percents / actual_percents)) return psi_value reference = np.random.normal(0, 1, 1000) new_data = np.random.normal(5, 1, 1000) print(psi(reference, new_data)) ```

A The PSI formula is incorrect and should use absolute differences

B The scale_range function modifies the input array in place causing incorrect binning

C The breakpoints array is not sorted, causing histogram errors

D The new_data and reference arrays have different lengths causing division errors

💡 Hint

Check if the input arrays are changed unexpectedly inside the function.

The scale_range function changes the input array directly, which affects the histogram calculation and leads to wrong PSI values.

Practice

1. What is the main purpose of model drift detection in machine learning?

easy

A. To check if the model's performance decreases over time

B. To increase the model's training speed

C. To reduce the size of the training dataset

D. To improve the model's accuracy during training

Solution

1. Step 1: Understand model drift concept

   Model drift means the model's predictions become less accurate over time due to changes in data or environment.

2. Step 2: Identify the purpose of detection

   Detecting drift helps know when the model needs retraining or updating to maintain performance.

3. Final Answer:

   To check if the model's performance decreases over time -> Option A

4. Quick Check:

   Model drift detection = performance drop check [OK]

Hint: Model drift means performance drops over time [OK]

Common Mistakes:

• Confusing drift detection with training speed improvement
• Thinking drift reduces dataset size
• Assuming drift improves accuracy during training

2. Which of the following is a correct Python syntax to compare model predictions before and after deployment for drift detection?

easy

A. drift = old_preds + new_preds

B. drift = compare(old_preds, new_preds)

C. drift = old_preds == new_preds

D. drift = np.mean(old_preds != new_preds)

Solution

1. Step 1: Understand drift comparison logic

   Drift detection often measures how much predictions differ, so counting mismatches is useful.

2. Step 2: Check syntax correctness

   drift = np.mean(old_preds != new_preds) uses numpy mean of mismatches (old_preds != new_preds), which correctly calculates difference ratio.

3. Final Answer:

   drift = np.mean(old_preds != new_preds) -> Option D

4. Quick Check:

   Correct syntax for prediction difference = drift = np.mean(old_preds != new_preds) [OK]

Hint: Use mean of mismatches to measure prediction drift [OK]

Common Mistakes:

• Using equality operator without aggregation
• Assuming a function 'compare' exists by default
• Adding predictions instead of comparing

3. Given the following code snippet for drift detection, what will be the output?

import numpy as np
old_preds = np.array([1, 0, 1, 1, 0])
new_preds = np.array([1, 1, 1, 0, 0])
drift_score = np.mean(old_preds != new_preds)
print(round(drift_score, 2))

medium

A. 0.40

B. 0.80

C. 0.60

D. 0.20

Solution

1. Step 1: Calculate mismatches between old_preds and new_preds

   Comparing element-wise: [1==1, 0!=1, 1==1, 1!=0, 0==0] gives mismatches at positions 1 and 3 (2 mismatches).

2. Step 2: Compute mean of mismatches

   Number of mismatches = 2, total elements = 5, so drift_score = 2/5 = 0.4.

3. Final Answer:

   0.40 -> Option A

4. Quick Check:

   Mismatch count / total = 0.4 [OK]

Hint: Count mismatches, divide by total elements [OK]

Common Mistakes:

• Counting matches instead of mismatches
• Rounding incorrectly
• Miscounting array length

4. Identify the error in this drift detection code snippet:

old_data = [0.1, 0.2, 0.3]
new_data = [0.15, 0.25, 0.35]
drift = sum(old_data != new_data) / len(old_data)
print(drift)

medium

A. len() function is missing parentheses

B. Using list comparison with != causes an error

C. sum() cannot be used on lists

D. No error, code runs correctly

Solution

1. Step 1: Understand list comparison behavior

   Using != between two lists compares entire lists, not element-wise, so old_data != new_data returns a single boolean, not element-wise array.

2. Step 2: Identify error cause

   sum() expects iterable of numbers, but gets a single boolean, causing a TypeError or wrong result.

3. Final Answer:

   Using list comparison with != causes an error -> Option B

4. Quick Check:

   List != list is not element-wise [OK]

Hint: Use numpy arrays for element-wise comparison [OK]

Common Mistakes:

• Thinking sum() can't be used on lists
• Forgetting parentheses in len()
• Assuming no error occurs

5. You have a deployed model and want to detect drift by comparing feature distributions before and after deployment. Which approach is best to detect drift in continuous features?

hard

A. Calculate accuracy difference between old and new predictions

B. Count how many predictions changed between old and new data

C. Use Kolmogorov-Smirnov test to compare old and new feature distributions

D. Retrain the model immediately without checking data

Solution

1. Step 1: Understand drift detection for continuous features

   Comparing distributions of continuous features requires statistical tests to detect changes in data patterns.

2. Step 2: Identify appropriate statistical test

   Kolmogorov-Smirnov test is a common method to compare two continuous distributions and detect drift.

3. Final Answer:

   Use Kolmogorov-Smirnov test to compare old and new feature distributions -> Option C

4. Quick Check:

   Statistical test for continuous feature drift = Use Kolmogorov-Smirnov test to compare old and new feature distributions [OK]

Hint: Use KS test for continuous feature distribution drift [OK]

Common Mistakes:

• Using accuracy difference for feature drift
• Counting prediction changes instead of feature changes
• Retraining without drift check