Why models degrade in production in MLOps - See It in Action

Understanding Why Models Degrade in Production

📖 Scenario: You are working as a machine learning engineer. You deployed a model to predict customer churn. After some time, the model's accuracy dropped. You want to understand why models degrade in production.

🎯 Goal: Build a simple Python script that simulates model performance degradation by comparing initial and new data accuracy scores. This will help you see how changes in data affect model quality over time.

📋 What You'll Learn

Create a dictionary called initial_accuracy with keys 'accuracy' and 'date' and values 0.85 and '2024-01-01'

Create a dictionary called new_accuracy with keys 'accuracy' and 'date' and values 0.65 and '2024-06-01'

Create a variable called degradation_threshold and set it to 0.1

Write an if statement that compares the accuracy drop between initial_accuracy['accuracy'] and new_accuracy['accuracy'] to the degradation_threshold

Print 'Model performance degraded significantly' if the drop is greater than the threshold, otherwise print 'Model performance is stable'

💡 Why This Matters

🌍 Real World

In real life, machine learning models often face changing data patterns after deployment. Monitoring accuracy helps detect when models need retraining.

💼 Career

Understanding model degradation is key for MLOps engineers to maintain reliable AI systems and ensure business decisions stay accurate.

1

Create initial and new accuracy data

Create a dictionary called initial_accuracy with keys 'accuracy' and 'date' and values 0.85 and '2024-01-01'. Also create a dictionary called new_accuracy with keys 'accuracy' and 'date' and values 0.65 and '2024-06-01'.

MLOps

# Create initial_accuracy and new_accuracy dictionaries
# Your code here

Need a hint?

Use curly braces {} to create dictionaries with the exact keys and values.

2

Set degradation threshold

Create a variable called degradation_threshold and set it to 0.1.

MLOps

initial_accuracy = {'accuracy': 0.85, 'date': '2024-01-01'}
new_accuracy = {'accuracy': 0.65, 'date': '2024-06-01'}
# Create degradation_threshold variable
# Your code here

Need a hint?

Use a simple assignment statement to create the threshold variable.

3

Check if model performance degraded

Write an if statement that compares the difference between initial_accuracy['accuracy'] and new_accuracy['accuracy'] to the degradation_threshold. Use variables initial_accuracy, new_accuracy, and degradation_threshold exactly.

MLOps

initial_accuracy = {'accuracy': 0.85, 'date': '2024-01-01'}
new_accuracy = {'accuracy': 0.65, 'date': '2024-06-01'}
degradation_threshold = 0.1
# Write if statement to check accuracy drop
# Your code here

Need a hint?

Calculate the difference and compare it to the threshold using an if-else statement.

4

Print the degradation result

Write a print statement to display the variable result.

MLOps

initial_accuracy = {'accuracy': 0.85, 'date': '2024-01-01'}
new_accuracy = {'accuracy': 0.65, 'date': '2024-06-01'}
degradation_threshold = 0.1

if (initial_accuracy['accuracy'] - new_accuracy['accuracy']) > degradation_threshold:
    result = 'Model performance degraded significantly'
else:
    result = 'Model performance is stable'

# Print the result
# Your code here

Need a hint?

Use print(result) to show the message.