Monitoring model performance in ML Python - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding model drift

What does model drift mean in the context of monitoring machine learning models?

A The model's input data distribution changes, causing performance degradation

B The model's performance improves over time without retraining

C The model is retrained with new data regularly

D The model's architecture is updated to a newer version

💡 Hint

Think about what happens when the data the model sees changes after deployment.

Model drift happens when the data the model receives changes from what it was trained on, leading to worse predictions.

💻 Command Output

intermediate

Interpreting monitoring logs

You run a monitoring script that outputs the following line:

Warning: Model accuracy dropped from 0.92 to 0.75 in last 24 hours

What does this output indicate?

A The model's accuracy has dropped significantly recently

B The model's accuracy has improved recently

C The model's accuracy stayed the same

D The monitoring script failed to run

💡 Hint

Look at the numbers and the word 'dropped'.

The output shows the accuracy decreased from 0.92 to 0.75, which means the model is performing worse.

🔀 Workflow

advanced

Setting up automated alerts for model performance

You want to create an automated alert that triggers when model precision falls below 0.8. Which step should you include in your monitoring workflow?

A Manually check model performance once a month

B Schedule daily retraining of the model regardless of performance

C Ignore precision and only monitor model latency

D Collect prediction results and calculate precision metrics regularly

💡 Hint

Think about what data you need to calculate precision and how often.

To alert on precision, you must collect prediction results and compute precision regularly to detect drops.

❓ Troubleshoot

advanced

Diagnosing sudden drop in model recall

Your monitoring dashboard shows a sudden drop in model recall. Which of the following is the most likely cause?

A The model's output labels were manually corrected

B The model's input data distribution has shifted

C The model's latency increased slightly

D The model's training code was updated but not deployed

💡 Hint

Recall measures how many true positives the model finds. What could affect this?

A shift in input data distribution can cause the model to miss more true positives, lowering recall.

✅ Best Practice

expert

Choosing metrics for model monitoring

Which metric is best to monitor for a fraud detection model where false negatives are very costly?

A Accuracy

B Latency

C Recall

D Precision

💡 Hint

False negatives mean fraud cases missed by the model. Which metric focuses on catching positives?

Recall measures how many actual fraud cases the model detects, so it's critical when missing fraud is costly.