Model drift detection in ML Python - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

What is model drift in machine learning?

Which of the following best describes model drift?

A When the data distribution changes causing the model's predictions to become less accurate

B When the model's performance improves over time without retraining

C When the model is trained on more data than originally planned

D When the model's architecture is changed during training

💡 Hint

Think about what happens if the world changes but the model stays the same.

Model drift happens when the data the model sees changes over time, making its predictions less accurate because it was trained on old data.

❓ Metrics

intermediate

Detecting model drift using metrics

You have a classification model deployed in production. Which metric change would most likely indicate model drift?

A Training loss decreases during model training

B Precision and recall remain constant

C Sudden drop in F1-score on new data compared to training data

D Accuracy increases steadily over time

💡 Hint

Model drift usually hurts performance on new data.

A sudden drop in F1-score on new data means the model is not predicting well anymore, which is a sign of drift.

❓ Predict Output

advanced

Output of drift detection code snippet

What is the output of this Python code that compares feature distributions to detect drift?

ML Python

import numpy as np
from scipy.stats import ks_2samp

reference = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9])
new_data = np.array([2, 3, 4, 5, 6, 7, 8, 9, 10])

stat, p_value = ks_2samp(reference, new_data)

if p_value < 0.05:
    print('Drift detected')
else:
    print('No drift detected')

A TypeError

B Drift detected

C SyntaxError

D No drift detected

💡 Hint

The Kolmogorov-Smirnov test checks if two samples come from the same distribution.

The p-value is above 0.05, so the test does not detect a significant difference between distributions.

❓ Hyperparameter

advanced

Choosing parameters for drift detection sensitivity

In a drift detection system using a statistical test, which parameter controls how sensitive the system is to detecting drift?

A The significance level (alpha) threshold for the test

B The number of features in the dataset

C The learning rate of the model

D The batch size of data processed

💡 Hint

This parameter decides how small the p-value must be to say drift exists.

The significance level (alpha) sets the cutoff p-value below which drift is declared, controlling sensitivity.

🔧 Debug

expert

Why does this drift detection code fail to detect drift?

Consider this code snippet for detecting drift using population stability index (PSI). Why does it fail to detect drift when new data distribution changes significantly? ```python import numpy as np def psi(expected, actual, buckets=10): def scale_range(input, min, max): input = input - np.min(input) input = input / np.max(input) * (max - min) input = input + min return input breakpoints = np.arange(0, buckets + 1) / buckets * 100 expected_percents = np.histogram(scale_range(expected, 0, 100), bins=breakpoints)[0] / len(expected) actual_percents = np.histogram(scale_range(actual, 0, 100), bins=breakpoints)[0] / len(actual) psi_value = np.sum((expected_percents - actual_percents) * np.log(expected_percents / actual_percents)) return psi_value reference = np.random.normal(0, 1, 1000) new_data = np.random.normal(5, 1, 1000) print(psi(reference, new_data)) ```

A The PSI formula is incorrect and should use absolute differences

B The scale_range function modifies the input array in place causing incorrect binning

C The breakpoints array is not sorted, causing histogram errors

D The new_data and reference arrays have different lengths causing division errors

💡 Hint

Check if the input arrays are changed unexpectedly inside the function.

The scale_range function changes the input array directly, which affects the histogram calculation and leads to wrong PSI values.