Prediction distribution monitoring in MLOps - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding prediction distribution monitoring purpose

What is the main goal of prediction distribution monitoring in an ML system?

A To detect changes in the prediction distribution that may affect model performance

B To track the number of API calls made to the prediction service

C To monitor the hardware usage of the ML deployment environment

D To optimize the training speed of the machine learning model

💡 Hint

Think about what can cause a model to perform worse after deployment.

Prediction distribution monitoring helps detect when the prediction distribution in production differs from training data, which can degrade performance.

💻 Command Output

intermediate

Interpreting output of a distribution drift detection tool

Given the following output from a drift detection tool monitoring prediction probabilities, what does it indicate?

{"drift_detected": true, "p_value": 0.01, "metric": "kolmogorov_smirnov"}

A The tool failed to run due to a syntax error

B The model predictions are exactly the same as baseline

C The prediction distribution has significantly changed compared to baseline

D The p_value indicates no significant change in distribution

💡 Hint

Recall that a low p-value means strong evidence against the null hypothesis.

A p-value of 0.01 means the distribution difference is statistically significant, so drift is detected.

❓ Configuration

advanced

Configuring a monitoring job for prediction distribution

Which configuration snippet correctly sets up a monitoring job to track prediction probability distribution using a Kolmogorov-Smirnov test every hour?

A

monitoring_job:
  frequency: every_minute
  metric: kolmogorov_smirnov
  data_source: model_weights
  alert_threshold: 0.05

B

monitoring_job:
  frequency: daily
  metric: accuracy
  data_source: input_features
  alert_threshold: 0.01

C

monitoring_job:
  frequency: hourly
  metric: mean_squared_error
  data_source: prediction_probabilities
  alert_threshold: 0.1

D

monitoring_job:
  frequency: hourly
  metric: kolmogorov_smirnov
  data_source: prediction_probabilities
  alert_threshold: 0.05

💡 Hint

Focus on frequency, metric type, and data source relevant to prediction distribution.

Option D correctly configures hourly monitoring of prediction probabilities using Kolmogorov-Smirnov test with a reasonable alert threshold.

❓ Troubleshoot

advanced

Diagnosing missing alerts in prediction distribution monitoring

An ML engineer notices no alerts are triggered despite clear changes in prediction distribution. Which is the most likely cause?

A The alert threshold is set too high, preventing alerts from triggering

B The prediction service is running on outdated hardware

C The monitoring job frequency is set to daily instead of hourly

D The model training data was too large

💡 Hint

Consider how alert thresholds affect sensitivity.

If the alert threshold is too high, small but important distribution changes won't trigger alerts.

🔀 Workflow

expert

Steps to implement prediction distribution monitoring in production

What is the correct order of steps to implement prediction distribution monitoring for a deployed ML model?

A 1,2,3,4

B 1,3,2,4

C 2,1,3,4

D 3,1,2,4

💡 Hint

Think about what you need before deploying monitoring and alerting.

First collect baseline data, then define tests and thresholds, next deploy monitoring, and finally set up alerts and dashboards.