Monitoring model performance in ML Python - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding model drift

What does model drift mean in the context of monitoring machine learning models?

A The model's input data distribution changes, causing performance degradation

B The model's performance improves over time without retraining

C The model is retrained with new data regularly

D The model's architecture is updated to a newer version

💡 Hint

Think about what happens when the data the model sees changes after deployment.

Model drift happens when the data the model receives changes from what it was trained on, leading to worse predictions.

💻 Command Output

intermediate

Interpreting monitoring logs

You run a monitoring script that outputs the following line:

Warning: Model accuracy dropped from 0.92 to 0.75 in last 24 hours

What does this output indicate?

A The model's accuracy has dropped significantly recently

B The model's accuracy has improved recently

C The model's accuracy stayed the same

D The monitoring script failed to run

💡 Hint

Look at the numbers and the word 'dropped'.

The output shows the accuracy decreased from 0.92 to 0.75, which means the model is performing worse.

🔀 Workflow

advanced

Setting up automated alerts for model performance

You want to create an automated alert that triggers when model precision falls below 0.8. Which step should you include in your monitoring workflow?

A Manually check model performance once a month

B Schedule daily retraining of the model regardless of performance

C Ignore precision and only monitor model latency

D Collect prediction results and calculate precision metrics regularly

💡 Hint

Think about what data you need to calculate precision and how often.

To alert on precision, you must collect prediction results and compute precision regularly to detect drops.

❓ Troubleshoot

advanced

Diagnosing sudden drop in model recall

Your monitoring dashboard shows a sudden drop in model recall. Which of the following is the most likely cause?

A The model's output labels were manually corrected

B The model's input data distribution has shifted

C The model's latency increased slightly

D The model's training code was updated but not deployed

💡 Hint

Recall measures how many true positives the model finds. What could affect this?

A shift in input data distribution can cause the model to miss more true positives, lowering recall.

✅ Best Practice

expert

Choosing metrics for model monitoring

Which metric is best to monitor for a fraud detection model where false negatives are very costly?

A Accuracy

B Latency

C Recall

D Precision

💡 Hint

False negatives mean fraud cases missed by the model. Which metric focuses on catching positives?

Recall measures how many actual fraud cases the model detects, so it's critical when missing fraud is costly.

Practice

1. Why is it important to monitor a machine learning model after deployment?

easy

A. To ensure the model continues to perform well on new data

B. To increase the model's training speed

C. To reduce the size of the model file

D. To change the model's algorithm automatically

Solution

1. Step 1: Understand model deployment context

   After deployment, models face new data that may differ from training data.

2. Step 2: Importance of monitoring

   Monitoring helps detect if the model's accuracy or other metrics drop over time.

3. Final Answer:

   To ensure the model continues to perform well on new data -> Option A

4. Quick Check:

   Monitoring = Ensure performance [OK]

Hint: Monitoring keeps models accurate after deployment [OK]

Common Mistakes:

• Confusing monitoring with training speed
• Thinking monitoring changes model algorithm automatically
• Assuming monitoring reduces model size

2. Which of the following is the correct way to calculate accuracy in Python given true labels y_true and predicted labels y_pred?

easy

A. accuracy = sum(y_true != y_pred) / len(y_true)

B. accuracy = len(y_true) / sum(y_true == y_pred)

C. accuracy = sum(y_true == y_pred) / len(y_true)

D. accuracy = sum(y_pred) / sum(y_true)

Solution

1. Step 1: Understand accuracy formula

   Accuracy = number of correct predictions divided by total predictions.

2. Step 2: Match formula to code

   sum(y_true == y_pred) counts correct predictions; dividing by len(y_true) gives accuracy.

3. Final Answer:

   accuracy = sum(y_true == y_pred) / len(y_true) -> Option C

4. Quick Check:

   Correct predictions / total = accuracy [OK]

Hint: Accuracy = correct predictions divided by total [OK]

Common Mistakes:

• Dividing total by correct instead of correct by total
• Using inequality instead of equality for correct count
• Summing predictions instead of comparing labels

3. Given the following Python code snippet monitoring model accuracy over time, what will be the printed output?

accuracies = [0.9, 0.85, 0.8, 0.75]
threshold = 0.8
alerts = [acc < threshold for acc in accuracies]
print(alerts)

medium

A. [False, False, True, True]

B. [True, True, True, True]

C. [True, False, False, False]

D. [False, False, False, True]

Solution

1. Step 1: Evaluate each accuracy against threshold

   Check if each accuracy is less than 0.8: 0.9 < 0.8? False; 0.85 < 0.8? False; 0.8 < 0.8? False; 0.75 < 0.8? True.

2. Step 2: Create alerts list

   List comprehension results in [False, False, False, True].

3. Final Answer:

   [False, False, False, True] -> Option D

4. Quick Check:

   Compare each value to threshold [OK]

Hint: Check each value against threshold in list comprehension [OK]

Common Mistakes:

• Misunderstanding < vs <= in comparisons
• Confusing True/False order in list
• Assuming 0.8 < 0.8 is True

4. You have this Python code to monitor model loss, but it raises an error. What is the error and how to fix it?

losses = [0.3, 0.25, 0.4, 0.35]
for i in range(len(losses))
    if losses[i] > 0.3:
        print(f"High loss at index {i}: {losses[i]}")

medium

A. SyntaxError due to missing colon after for loop; add ':' after 'range(len(losses))' to fix

B. TypeError because losses is not iterable; convert losses to list to fix

C. IndentationError due to wrong indentation; fix indentation of print statement

D. NameError because variable 'i' is undefined; define 'i' before loop

Solution

1. Step 1: Identify syntax error in for loop

   The for loop line is missing a colon at the end, which is required in Python syntax.

2. Step 2: Fix syntax by adding colon

   Add ':' after range(len(losses)) to correct the syntax and allow the loop to run.

3. Final Answer:

   SyntaxError due to missing colon after for loop; add ':' after 'range(len(losses))' to fix -> Option A

4. Quick Check:

   Missing colon = SyntaxError [OK]

Hint: Check colons after for/if statements [OK]

Common Mistakes:

• Ignoring missing colon causing syntax error
• Confusing indentation error with syntax error
• Assuming variable 'i' is undefined

5. You want to automate alerts when your model's F1 score drops below 0.7. Which approach best fits this monitoring task?

hard

A. Ignore F1 score and monitor only training loss during training

B. Set up a scheduled job that calculates F1 score daily and sends an alert if below 0.7

C. Manually check F1 score once a month and update the model if needed

D. Retrain the model automatically every hour regardless of performance

Solution

1. Step 1: Understand monitoring automation

   Automated scheduled jobs can regularly check model metrics like F1 score.

2. Step 2: Choose alerting based on threshold

   Sending alerts when F1 score drops below 0.7 helps catch performance issues early.

3. Final Answer:

   Set up a scheduled job that calculates F1 score daily and sends an alert if below 0.7 -> Option B

4. Quick Check:

   Automate checks + alerts = Set up a scheduled job that calculates F1 score daily and sends an alert if below 0.7 [OK]

Hint: Automate metric checks and alert on threshold breach [OK]

Common Mistakes:

• Retraining without checking performance first
• Manual checks too infrequent for timely alerts
• Ignoring key metrics like F1 score