Concept drift detection in MLOps - Mini Project: Build & Apply

Concept Drift Detection in Machine Learning

📖 Scenario: You are working as a machine learning engineer in a company that deploys models to predict customer behavior. Over time, the data your model sees can change, which may cause the model to perform worse. This change is called concept drift. Detecting concept drift early helps keep the model accurate and reliable.

🎯 Goal: Build a simple Python program that detects concept drift by comparing the distribution of new data with the original training data using a threshold.

📋 What You'll Learn

Create a dictionary called training_data_distribution with exact counts for categories

Create a variable called drift_threshold with the exact value 0.2

Write a function called detect_drift that takes two dictionaries and returns true if drift is detected

Print the result of calling detect_drift with training_data_distribution and new_data_distribution

💡 Why This Matters

🌍 Real World

Concept drift detection is crucial in real-world machine learning systems where data changes over time, such as fraud detection, recommendation systems, and customer behavior prediction.

💼 Career

Understanding and implementing concept drift detection helps machine learning engineers and MLOps professionals maintain model accuracy and reliability in production environments.

1

Create the training data distribution

Create a dictionary called training_data_distribution with these exact entries: 'A': 50, 'B': 30, 'C': 20.

MLOps

# Create the training data distribution dictionary
# Your code here

Hint

Use curly braces {} to create a dictionary with keys 'A', 'B', and 'C' and their counts.

2

Set the drift detection threshold

Create a variable called drift_threshold and set it to the float value 0.2.

MLOps

training_data_distribution = {'A': 50, 'B': 30, 'C': 20}
# Set the drift detection threshold
# Your code here

Hint

Assign the value 0.2 to the variable drift_threshold.

3

Write the concept drift detection function

Write a function called detect_drift that takes two dictionaries: original and new. It should calculate the total absolute difference in proportions for keys 'A', 'B', and 'C'. Return true if this difference is greater than or equal to drift_threshold, otherwise false. Use the formula: difference = sum of absolute differences of (new[key]/new_total) and (original[key]/original_total) for each key.

MLOps

training_data_distribution = {'A': 50, 'B': 30, 'C': 20}
drift_threshold = 0.2

# Write the detect_drift function below
# Your code here

Hint

Calculate proportions by dividing counts by total counts. Sum absolute differences. Compare with drift_threshold.

4

Test and print the drift detection result

Create a dictionary called new_data_distribution with these exact entries: 'A': 40, 'B': 35, 'C': 25. Then print the result of calling detect_drift(training_data_distribution, new_data_distribution).

MLOps

training_data_distribution = {'A': 50, 'B': 30, 'C': 20}
drift_threshold = 0.2

def detect_drift(original, new):
    original_total = sum(original.values())
    new_total = sum(new.values())
    difference = 0
    for key in ['A', 'B', 'C']:
        original_prop = original[key] / original_total
        new_prop = new[key] / new_total
        difference += abs(new_prop - original_prop)
    return difference >= drift_threshold

# Create new_data_distribution and print the drift detection result
# Your code here

Hint

Use the exact dictionary for new_data_distribution. Call detect_drift with the two dictionaries and print the result.

Practice

1. What is the main purpose of concept drift detection in machine learning?

easy

A. To identify when the data distribution changes over time affecting model accuracy

B. To increase the training speed of a machine learning model

C. To reduce the size of the training dataset

D. To improve the hardware performance for model training

Solution

1. Step 1: Understand concept drift meaning

   Concept drift means the data changes over time, causing model accuracy to drop.

2. Step 2: Identify the purpose of detection

   Detecting drift helps know when the model needs updating to keep accuracy high.

3. Final Answer:

   To identify when the data distribution changes over time affecting model accuracy -> Option A

4. Quick Check:

   Concept drift detection = find data changes [OK]

Hint: Concept drift means data changes; detection finds these changes [OK]

Common Mistakes:

• Confusing drift detection with speeding up training
• Thinking drift reduces dataset size
• Assuming drift improves hardware

2. Which of the following is a correct method to detect concept drift?

easy

A. Reduce the number of model layers

B. Increase the batch size during model training

C. Use a larger learning rate

D. Compare model accuracy on recent data versus older data

Solution

1. Step 1: Identify drift detection methods

   Drift detection compares model performance on new data to old data to find changes.

2. Step 2: Evaluate options

   Only comparing accuracy over time relates to drift detection; others affect training but not drift.

3. Final Answer:

   Compare model accuracy on recent data versus older data -> Option D

4. Quick Check:

   Drift detection = compare old vs new accuracy [OK]

Hint: Drift detection compares model accuracy over time [OK]

Common Mistakes:

• Confusing training hyperparameters with drift detection
• Thinking batch size or learning rate detect drift
• Ignoring performance comparison over time

3. Given this Python snippet for drift detection:

old_accuracy = 0.85
new_accuracy = 0.70
threshold = 0.1
if old_accuracy - new_accuracy > threshold:
    print('Drift detected')
else:
    print('No drift')

What will be the output?

medium

A. No drift

B. SyntaxError

C. Drift detected

D. No output

Solution

1. Step 1: Calculate accuracy difference

   old_accuracy - new_accuracy = 0.85 - 0.70 = 0.15

2. Step 2: Compare difference to threshold

   0.15 > 0.1, so condition is true and 'Drift detected' prints.

3. Final Answer:

   Drift detected -> Option C

4. Quick Check:

   0.15 > 0.1 means drift detected [OK]

Hint: Subtract accuracies and compare to threshold [OK]

Common Mistakes:

• Mixing up greater than and less than signs
• Ignoring the threshold value
• Assuming syntax error due to > symbol

4. This code snippet is intended to detect concept drift but has an error:

old_acc = 0.9
new_acc = 0.85
threshold = 0.05
if new_acc - old_acc > threshold:
    print('Drift detected')
else:
    print('No drift')

What is the error?

medium

A. The threshold value is too low to detect drift

B. The condition should be 'old_acc - new_acc > threshold' to detect accuracy drop

C. The print statements are reversed

D. There is a syntax error in the if statement

Solution

1. Step 1: Understand drift detection logic

   Drift means accuracy drops, so we check if old accuracy minus new accuracy exceeds threshold.

2. Step 2: Analyze the condition

   The code checks if new_acc - old_acc > threshold, which is negative here (0.85 - 0.9 = -0.05), so it won't detect drift correctly.

3. Final Answer:

   The condition should be 'old_acc - new_acc > threshold' to detect accuracy drop -> Option B

4. Quick Check:

   Check accuracy drop as old - new > threshold [OK]

Hint: Subtract new accuracy from old to detect drop [OK]

Common Mistakes:

• Subtracting in wrong order
• Assuming threshold value causes error
• Thinking print statements cause problem

5. You have a model deployed in production. You want to detect concept drift using data distribution changes. Which approach is best to implement?

hard

A. Monitor statistical differences in feature distributions between training and recent data

B. Retrain the model daily regardless of data changes

C. Only monitor model accuracy without checking data

D. Increase model complexity to handle all data variations

Solution

1. Step 1: Understand concept drift detection methods

   Detecting drift by monitoring data distribution changes helps catch shifts before accuracy drops.

2. Step 2: Evaluate options for best practice

   Monitor statistical differences in feature distributions between training and recent data uses statistical tests on features, which is a proactive and effective drift detection method. Other options either ignore data changes or waste resources.

3. Final Answer:

   Monitor statistical differences in feature distributions between training and recent data -> Option A

4. Quick Check:

   Data distribution monitoring = best drift detection [OK]

Hint: Check feature stats differences to detect drift early [OK]

Common Mistakes:

• Retraining blindly without drift detection
• Ignoring data changes and only watching accuracy
• Assuming bigger models fix drift automatically