Why models degrade in production in MLOps - See It in Action

Understanding Why Models Degrade in Production

📖 Scenario: You are working as a machine learning engineer. You deployed a model to predict customer churn. After some time, the model's accuracy dropped. You want to understand why models degrade in production.

🎯 Goal: Build a simple Python script that simulates model performance degradation by comparing initial and new data accuracy scores. This will help you see how changes in data affect model quality over time.

📋 What You'll Learn

Create a dictionary called initial_accuracy with keys 'accuracy' and 'date' and values 0.85 and '2024-01-01'

Create a dictionary called new_accuracy with keys 'accuracy' and 'date' and values 0.65 and '2024-06-01'

Create a variable called degradation_threshold and set it to 0.1

Write an if statement that compares the accuracy drop between initial_accuracy['accuracy'] and new_accuracy['accuracy'] to the degradation_threshold

Print 'Model performance degraded significantly' if the drop is greater than the threshold, otherwise print 'Model performance is stable'

💡 Why This Matters

🌍 Real World

In real life, machine learning models often face changing data patterns after deployment. Monitoring accuracy helps detect when models need retraining.

💼 Career

Understanding model degradation is key for MLOps engineers to maintain reliable AI systems and ensure business decisions stay accurate.

1

Create initial and new accuracy data

Create a dictionary called initial_accuracy with keys 'accuracy' and 'date' and values 0.85 and '2024-01-01'. Also create a dictionary called new_accuracy with keys 'accuracy' and 'date' and values 0.65 and '2024-06-01'.

MLOps

# Create initial_accuracy and new_accuracy dictionaries
# Your code here

Hint

Use curly braces {} to create dictionaries with the exact keys and values.

2

Set degradation threshold

Create a variable called degradation_threshold and set it to 0.1.

MLOps

initial_accuracy = {'accuracy': 0.85, 'date': '2024-01-01'}
new_accuracy = {'accuracy': 0.65, 'date': '2024-06-01'}
# Create degradation_threshold variable
# Your code here

Hint

Use a simple assignment statement to create the threshold variable.

3

Check if model performance degraded

Write an if statement that compares the difference between initial_accuracy['accuracy'] and new_accuracy['accuracy'] to the degradation_threshold. Use variables initial_accuracy, new_accuracy, and degradation_threshold exactly.

MLOps

initial_accuracy = {'accuracy': 0.85, 'date': '2024-01-01'}
new_accuracy = {'accuracy': 0.65, 'date': '2024-06-01'}
degradation_threshold = 0.1
# Write if statement to check accuracy drop
# Your code here

Hint

Calculate the difference and compare it to the threshold using an if-else statement.

4

Print the degradation result

Write a print statement to display the variable result.

MLOps

initial_accuracy = {'accuracy': 0.85, 'date': '2024-01-01'}
new_accuracy = {'accuracy': 0.65, 'date': '2024-06-01'}
degradation_threshold = 0.1

if (initial_accuracy['accuracy'] - new_accuracy['accuracy']) > degradation_threshold:
    result = 'Model performance degraded significantly'
else:
    result = 'Model performance is stable'

# Print the result
# Your code here

Hint

Use print(result) to show the message.

Practice

1. Why do machine learning models often degrade when deployed in production?

easy

A. Because the model code is always incorrect

B. Because the data or environment changes over time

C. Because production servers are slower

D. Because models never work outside training

Solution

1. Step 1: Understand model dependency on data

   Models learn patterns from training data, so if data changes, predictions may worsen.

2. Step 2: Recognize environment changes

   Changes in user behavior or system environment can cause model performance to drop.

3. Final Answer:

   Because the data or environment changes over time -> Option B

4. Quick Check:

   Model degradation = data/environment change [OK]

Hint: Models degrade when input data changes from training data [OK]

Common Mistakes:

• Thinking model code is always wrong
• Blaming server speed for model errors
• Assuming models never work outside training

2. Which of the following is a correct way to monitor model degradation in production?

easy

A. Stop collecting new data after deployment

B. Ignore model outputs and trust initial accuracy

C. Only retrain the model once a year

D. Track model performance metrics regularly

Solution

1. Step 1: Identify monitoring best practice

   Regularly tracking metrics like accuracy or error helps detect degradation early.

2. Step 2: Eliminate poor practices

   Ignoring outputs or stopping data collection prevents noticing problems timely.

3. Final Answer:

   Track model performance metrics regularly -> Option D

4. Quick Check:

   Monitoring = track metrics regularly [OK]

Hint: Monitor metrics often to catch degradation early [OK]

Common Mistakes:

• Ignoring model outputs after deployment
• Waiting too long to retrain
• Stopping data collection

3. Consider this code snippet monitoring model accuracy over time:

accuracies = [0.95, 0.93, 0.88, 0.85, 0.80]
if accuracies[-1] < 0.85:
    alert = True
else:
    alert = False
print(alert)

What will be the output and what does it indicate?

medium

A. True; model accuracy dropped below threshold

B. False; model accuracy is stable

C. Error; syntax mistake in code

D. True; model accuracy improved

Solution

1. Step 1: Check last accuracy value

   The last accuracy is 0.80, which is less than 0.85 threshold.

2. Step 2: Evaluate condition and output

   Since 0.80 < 0.85, alert is set to True and printed.

3. Final Answer:

   True; model accuracy dropped below threshold -> Option A

4. Quick Check:

   Last accuracy < threshold = True alert [OK]

Hint: Check last accuracy value against threshold [OK]

Common Mistakes:

• Confusing less than with greater than
• Assuming code has syntax error
• Thinking True means improvement

4. You have this monitoring code snippet:

accuracy = 0.82
if accuracy <= 0.8:
    print("Retrain model")
else:
    print("Model OK")

But the model is degrading and you want retraining to trigger at 0.85 accuracy or below. What is the fix?

medium

A. Remove else block

B. Change print statements order

C. Change condition to accuracy <= 0.85

D. Change accuracy variable to 0.9

Solution

1. Step 1: Identify current threshold

   Current code triggers retrain only if accuracy is 0.8 or less.

2. Step 2: Adjust threshold to 0.85

   Change condition to accuracy <= 0.85 to retrain earlier.

3. Final Answer:

   Change condition to accuracy <= 0.85 -> Option C

4. Quick Check:

   Retrain threshold = 0.85 [OK]

Hint: Update condition threshold to desired retrain point [OK]

Common Mistakes:

• Changing print order doesn't affect logic
• Removing else block won't fix threshold
• Changing accuracy value ignores real data

5. A deployed model's accuracy drops because the input data distribution changed. Which combined approach best addresses this degradation?

hard

A. Monitor performance, retrain with new data, and update deployment

B. Switch to a simpler model without monitoring

C. Retrain only when users complain

D. Ignore changes and keep using the model

Solution

1. Step 1: Recognize need for monitoring

   Monitoring detects when model accuracy drops due to data changes.

2. Step 2: Retrain and update model

   Retraining with new data adapts model to current distribution; redeploy updated model.

3. Final Answer:

   Monitor performance, retrain with new data, and update deployment -> Option A

4. Quick Check:

   Monitor + retrain + update = best practice [OK]

Hint: Combine monitoring, retraining, and deployment updates [OK]

Common Mistakes:

• Ignoring data changes
• Waiting for complaints before retraining
• Dropping monitoring leads to unnoticed failures