Model drift detection in ML Python - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to import the library used for detecting model drift.

ML Python

from sklearn.metrics import [1]

Drag options to blanks, or click blank then click option'

Amean_squared_error

Baccuracy_score

Cclassification_report

Droc_auc_score

Attempts:

3 left

💡 Hint

Common Mistakes

Choosing a metric that measures error instead of accuracy.

Importing a report function instead of a metric.

The accuracy_score function helps measure how well a model performs, which is useful for detecting model drift by comparing performance over time.

2fill in blank

medium

Complete the code to calculate the accuracy of model predictions.

ML Python

accuracy = accuracy_score(y_true, [1])

Drag options to blanks, or click blank then click option'

Ay_prob

By_test

Cy_train

Dy_pred

Attempts:

3 left

💡 Hint

Common Mistakes

Using predicted probabilities instead of predicted labels.

Confusing test data with predictions.

The accuracy_score function compares the true labels y_true with the predicted labels y_pred to calculate accuracy.

3fill in blank

hard

Fix the error in the code to detect model drift by comparing old and new accuracy.

ML Python

if new_accuracy [1] old_accuracy - threshold:
    print('Model drift detected')

Drag options to blanks, or click blank then click option'

A>

B==

C<

D>=

Attempts:

3 left

💡 Hint

Common Mistakes

Using greater than operator which means performance improved.

Using equality operator which doesn't detect drift.

Model drift is detected when the new accuracy is less than the old accuracy minus a threshold, indicating performance dropped.

4fill in blank

hard

Fill both blanks to create a dictionary that stores accuracy scores for old and new data.

ML Python

accuracy_scores = {'old': [1], 'new': [2]

Drag options to blanks, or click blank then click option'

Aold_accuracy

Bnew_accuracy

Caccuracy_score(y_old_true, y_old_pred)

Daccuracy_score(y_new_true, y_new_pred)

Attempts:

3 left

💡 Hint

Common Mistakes

Calling accuracy_score inside the dictionary instead of using variables.

Swapping old and new accuracy values.

The dictionary stores the pre-calculated accuracy values old_accuracy and new_accuracy for easy comparison.

5fill in blank

hard

Fill all three blanks to complete the code that calculates accuracy for old and new data and checks for drift.

ML Python

old_accuracy = [1](y_old_true, y_old_pred)
new_accuracy = [2](y_new_true, y_new_pred)
if new_accuracy [3] old_accuracy - 0.05:
    print('Model drift detected')

Drag options to blanks, or click blank then click option'

Aaccuracy_score

C<

D>

Attempts:

3 left

💡 Hint

Common Mistakes

Using different functions for accuracy calculation.

Using greater than operator which means performance improved.

Both accuracies are calculated using accuracy_score. The condition checks if new accuracy dropped below old accuracy minus 0.05, signaling drift.

Practice

1. What is the main purpose of model drift detection in machine learning?

easy

A. To check if the model's performance decreases over time

B. To increase the model's training speed

C. To reduce the size of the training dataset

D. To improve the model's accuracy during training

Solution

1. Step 1: Understand model drift concept

   Model drift means the model's predictions become less accurate over time due to changes in data or environment.

2. Step 2: Identify the purpose of detection

   Detecting drift helps know when the model needs retraining or updating to maintain performance.

3. Final Answer:

   To check if the model's performance decreases over time -> Option A

4. Quick Check:

   Model drift detection = performance drop check [OK]

Hint: Model drift means performance drops over time [OK]

Common Mistakes:

• Confusing drift detection with training speed improvement
• Thinking drift reduces dataset size
• Assuming drift improves accuracy during training

2. Which of the following is a correct Python syntax to compare model predictions before and after deployment for drift detection?

easy

A. drift = old_preds + new_preds

B. drift = compare(old_preds, new_preds)

C. drift = old_preds == new_preds

D. drift = np.mean(old_preds != new_preds)

Solution

1. Step 1: Understand drift comparison logic

   Drift detection often measures how much predictions differ, so counting mismatches is useful.

2. Step 2: Check syntax correctness

   drift = np.mean(old_preds != new_preds) uses numpy mean of mismatches (old_preds != new_preds), which correctly calculates difference ratio.

3. Final Answer:

   drift = np.mean(old_preds != new_preds) -> Option D

4. Quick Check:

   Correct syntax for prediction difference = drift = np.mean(old_preds != new_preds) [OK]

Hint: Use mean of mismatches to measure prediction drift [OK]

Common Mistakes:

• Using equality operator without aggregation
• Assuming a function 'compare' exists by default
• Adding predictions instead of comparing

3. Given the following code snippet for drift detection, what will be the output?

import numpy as np
old_preds = np.array([1, 0, 1, 1, 0])
new_preds = np.array([1, 1, 1, 0, 0])
drift_score = np.mean(old_preds != new_preds)
print(round(drift_score, 2))

medium

A. 0.40

B. 0.80

C. 0.60

D. 0.20

Solution

1. Step 1: Calculate mismatches between old_preds and new_preds

   Comparing element-wise: [1==1, 0!=1, 1==1, 1!=0, 0==0] gives mismatches at positions 1 and 3 (2 mismatches).

2. Step 2: Compute mean of mismatches

   Number of mismatches = 2, total elements = 5, so drift_score = 2/5 = 0.4.

3. Final Answer:

   0.40 -> Option A

4. Quick Check:

   Mismatch count / total = 0.4 [OK]

Hint: Count mismatches, divide by total elements [OK]

Common Mistakes:

• Counting matches instead of mismatches
• Rounding incorrectly
• Miscounting array length

4. Identify the error in this drift detection code snippet:

old_data = [0.1, 0.2, 0.3]
new_data = [0.15, 0.25, 0.35]
drift = sum(old_data != new_data) / len(old_data)
print(drift)

medium

A. len() function is missing parentheses

B. Using list comparison with != causes an error

C. sum() cannot be used on lists

D. No error, code runs correctly

Solution

1. Step 1: Understand list comparison behavior

   Using != between two lists compares entire lists, not element-wise, so old_data != new_data returns a single boolean, not element-wise array.

2. Step 2: Identify error cause

   sum() expects iterable of numbers, but gets a single boolean, causing a TypeError or wrong result.

3. Final Answer:

   Using list comparison with != causes an error -> Option B

4. Quick Check:

   List != list is not element-wise [OK]

Hint: Use numpy arrays for element-wise comparison [OK]

Common Mistakes:

• Thinking sum() can't be used on lists
• Forgetting parentheses in len()
• Assuming no error occurs

5. You have a deployed model and want to detect drift by comparing feature distributions before and after deployment. Which approach is best to detect drift in continuous features?

hard

A. Calculate accuracy difference between old and new predictions

B. Count how many predictions changed between old and new data

C. Use Kolmogorov-Smirnov test to compare old and new feature distributions

D. Retrain the model immediately without checking data

Solution

1. Step 1: Understand drift detection for continuous features

   Comparing distributions of continuous features requires statistical tests to detect changes in data patterns.

2. Step 2: Identify appropriate statistical test

   Kolmogorov-Smirnov test is a common method to compare two continuous distributions and detect drift.

3. Final Answer:

   Use Kolmogorov-Smirnov test to compare old and new feature distributions -> Option C

4. Quick Check:

   Statistical test for continuous feature drift = Use Kolmogorov-Smirnov test to compare old and new feature distributions [OK]

Hint: Use KS test for continuous feature distribution drift [OK]

Common Mistakes:

• Using accuracy difference for feature drift
• Counting prediction changes instead of feature changes
• Retraining without drift check