Prediction distribution monitoring in MLOps - Step-by-Step Execution

Process Flow - Prediction distribution monitoring

Collect Predictions

↓

Calculate Distribution

↓

Compare with Baseline

↓

Detect Drift?

No→Continue Monitoring

Yes↓

Alert & Investigate

↓

Take Action (Retrain/Adjust)

This flow shows how prediction outputs are collected, their distribution is calculated and compared to a baseline, and if drift is detected, alerts are triggered for investigation and action.

Execution Sample

MLOps

predictions = [0.1, 0.4, 0.35, 0.8, 0.9]
baseline_mean = 0.5
current_mean = sum(predictions)/len(predictions)
drift = abs(current_mean - baseline_mean) > 0.2
alert = drift

This code calculates the mean of current predictions, compares it to a baseline mean, and sets an alert if the difference is large.

Process Table

Step	Action	Value/Calculation	Result/State
1	Collect predictions	predictions = [0.1, 0.4, 0.35, 0.8, 0.9]	predictions list created
2	Calculate current mean	sum(predictions)/len(predictions) = (0.1+0.4+0.35+0.8+0.9)/5	0.51
3	Compare with baseline	abs(0.51 - 0.5) = 0.01	Difference = 0.01
4	Check if difference > 0.2	0.01 > 0.2	False
5	Set alert	alert = False	No drift detected, alert is False

💡 Difference between current and baseline mean is 0.01, which is not greater than 0.2, so no drift alert is triggered.

Status Tracker

Variable	Start	After Step 2	After Step 3	After Step 4	Final
predictions	undefined	[0.1, 0.4, 0.35, 0.8, 0.9]	[0.1, 0.4, 0.35, 0.8, 0.9]	[0.1, 0.4, 0.35, 0.8, 0.9]	[0.1, 0.4, 0.35, 0.8, 0.9]
current_mean	undefined	0.51	0.51	0.51	0.51
baseline_mean	0.5	0.5	0.5	0.5	0.5
drift	undefined	undefined	False	False	False
alert	undefined	undefined	undefined	False	False

Key Moments - 3 Insights

Why is the alert not triggered even though the current mean is different from the baseline?

Because the difference (0.01) is less than the threshold (0.2) as shown in step 4 of the execution table, so the condition to trigger alert is False.

What does the 'drift' variable represent in this process?

'drift' is a boolean that tells if the prediction distribution has changed significantly compared to baseline. It is False here because the difference is small (step 3 and 4).

Why do we calculate the mean of predictions instead of checking individual values?

The mean summarizes the overall distribution shift simply. Checking individual values would be noisy and less informative for drift detection.

Visual Quiz - 3 Questions

Look at the execution table, what is the value of current_mean at step 2?

A 0.5

B 0.01

C 0.51

D 0.2

💡 Hint

Check the 'Value/Calculation' and 'Result/State' columns at step 2 in the execution table.

At which step does the code decide that no drift has occurred?

A Step 4

B Step 5

C Step 3

D Step 2

💡 Hint

Look at the 'Check if difference > 0.2' condition and its result in the execution table.

If the baseline_mean was 0.3 instead of 0.5, what would be the alert value at the end?

A False

B True

C Undefined

D Error

💡 Hint

Calculate abs(current_mean - new baseline_mean) and compare with threshold >0.2 using variable_tracker values.

Concept Snapshot

Prediction distribution monitoring:
- Collect model predictions over time
- Calculate a summary statistic (e.g., mean)
- Compare with baseline distribution
- If difference exceeds threshold, flag drift
- Trigger alerts for investigation and action

Full Transcript

Prediction distribution monitoring means watching how the model's output predictions change over time. We collect predictions, calculate their average, and compare it to a baseline average from past data. If the difference is bigger than a set limit, we say drift happened and alert the team. This helps catch when the model might be less accurate due to changes in data or environment.