Prediction distribution monitoring in MLOps - Commands & Configuration

Introduction

Prediction distribution monitoring helps you watch how the results from your machine learning model change over time. It solves the problem of models becoming less accurate because the data they see in real life changes from what they were trained on.

When you want to check if your model's predictions are drifting away from what it learned during training

When you deploy a model and want to ensure it keeps making reliable predictions over weeks or months

When you want to detect sudden changes in the type of data your model is seeing, which might mean a problem

When you want to compare the distribution of new predictions to the original training predictions

When you want to alert your team if the model's prediction patterns change significantly

Commands

This command installs the Evidently library, which helps monitor prediction distributions easily.

Terminal

pip install evidently

Expected Output

Collecting evidently

  Downloading evidently-0.2.45-py3-none-any.whl (200 kB)

Installing collected packages: evidently

Successfully installed evidently-0.2.45

Runs a Python script that loads prediction data, compares current predictions to baseline, and prints a report on distribution changes.

Terminal

python monitor_predictions.py

Expected Output

Prediction distribution report:

- Kolmogorov-Smirnov test p-value: 0.03

- Warning: Prediction distribution has shifted significantly

- Mean prediction changed from 0.45 to 0.60

Key Concept

If you remember nothing else from this pattern, remember: monitoring prediction distributions helps catch when your model's outputs start to drift from what it learned, so you can fix problems early.

Code Example

MLOps

from evidently import ColumnMapping
from evidently.dashboard import Dashboard
from evidently.dashboard.tabs import DataDriftTab
import pandas as pd

# Load baseline and current prediction data
baseline = pd.read_csv('baseline_predictions.csv')
current = pd.read_csv('current_predictions.csv')

# Define which column holds predictions
column_mapping = ColumnMapping(prediction='prediction')

# Create a dashboard to check data drift in predictions
dashboard = Dashboard(tabs=[DataDriftTab()])
dashboard.calculate(baseline, current, column_mapping=column_mapping)

# Save the report as an HTML file
dashboard.save('prediction_drift_report.html')
print('Prediction drift report saved as prediction_drift_report.html')

Common Mistakes

Not comparing current predictions to a baseline distribution

Without a baseline, you cannot tell if the prediction distribution has changed or not.

Always save and use the original training or validation prediction distribution as a baseline for comparison.

Ignoring small but consistent changes in prediction distribution

Small changes can accumulate and degrade model performance over time if not addressed.

Set thresholds to detect even small shifts and monitor trends regularly.

Summary

Install the Evidently library to help monitor prediction distributions.

Run a script that compares current model predictions to a baseline to detect distribution changes.

Use a dashboard report to visualize and understand prediction drift over time.