Prediction distribution monitoring in MLOps - Step-by-Step Execution

Process Flow - Prediction distribution monitoring

Collect Predictions

↓

Calculate Distribution

↓

Compare with Baseline

↓

Detect Drift?

No→Continue Monitoring

Yes↓

Alert & Investigate

↓

Take Action (Retrain/Adjust)

This flow shows how prediction outputs are collected, their distribution is calculated and compared to a baseline, and if drift is detected, alerts are triggered for investigation and action.

Execution Sample

MLOps

predictions = [0.1, 0.4, 0.35, 0.8, 0.9]
baseline_mean = 0.5
current_mean = sum(predictions)/len(predictions)
drift = abs(current_mean - baseline_mean) > 0.2
alert = drift

This code calculates the mean of current predictions, compares it to a baseline mean, and sets an alert if the difference is large.

Process Table

Step	Action	Value/Calculation	Result/State
1	Collect predictions	predictions = [0.1, 0.4, 0.35, 0.8, 0.9]	predictions list created
2	Calculate current mean	sum(predictions)/len(predictions) = (0.1+0.4+0.35+0.8+0.9)/5	0.51
3	Compare with baseline	abs(0.51 - 0.5) = 0.01	Difference = 0.01
4	Check if difference > 0.2	0.01 > 0.2	False
5	Set alert	alert = False	No drift detected, alert is False

💡 Difference between current and baseline mean is 0.01, which is not greater than 0.2, so no drift alert is triggered.

Status Tracker

Variable	Start	After Step 2	After Step 3	After Step 4	Final
predictions	undefined	[0.1, 0.4, 0.35, 0.8, 0.9]	[0.1, 0.4, 0.35, 0.8, 0.9]	[0.1, 0.4, 0.35, 0.8, 0.9]	[0.1, 0.4, 0.35, 0.8, 0.9]
current_mean	undefined	0.51	0.51	0.51	0.51
baseline_mean	0.5	0.5	0.5	0.5	0.5
drift	undefined	undefined	False	False	False
alert	undefined	undefined	undefined	False	False

Key Moments - 3 Insights

Why is the alert not triggered even though the current mean is different from the baseline?

Because the difference (0.01) is less than the threshold (0.2) as shown in step 4 of the execution table, so the condition to trigger alert is False.

What does the 'drift' variable represent in this process?

'drift' is a boolean that tells if the prediction distribution has changed significantly compared to baseline. It is False here because the difference is small (step 3 and 4).

Why do we calculate the mean of predictions instead of checking individual values?

The mean summarizes the overall distribution shift simply. Checking individual values would be noisy and less informative for drift detection.

Visual Quiz - 3 Questions

Look at the execution table, what is the value of current_mean at step 2?

A 0.5

B 0.01

C 0.51

D 0.2

💡 Hint

Check the 'Value/Calculation' and 'Result/State' columns at step 2 in the execution table.

At which step does the code decide that no drift has occurred?

A Step 4

B Step 5

C Step 3

D Step 2

💡 Hint

Look at the 'Check if difference > 0.2' condition and its result in the execution table.

If the baseline_mean was 0.3 instead of 0.5, what would be the alert value at the end?

A False

B True

C Undefined

D Error

💡 Hint

Calculate abs(current_mean - new baseline_mean) and compare with threshold >0.2 using variable_tracker values.

Concept Snapshot

Prediction distribution monitoring:
- Collect model predictions over time
- Calculate a summary statistic (e.g., mean)
- Compare with baseline distribution
- If difference exceeds threshold, flag drift
- Trigger alerts for investigation and action

Full Transcript

Prediction distribution monitoring means watching how the model's output predictions change over time. We collect predictions, calculate their average, and compare it to a baseline average from past data. If the difference is bigger than a set limit, we say drift happened and alert the team. This helps catch when the model might be less accurate due to changes in data or environment.

Practice

1. What is the main purpose of prediction distribution monitoring in MLOps?

easy

A. To monitor the training data quality only

B. To track changes in the model's output predictions over time

C. To improve the speed of model training

D. To increase the size of the prediction dataset

Solution

1. Step 1: Understand prediction distribution monitoring

   It focuses on watching the outputs (predictions) of a model to detect changes or shifts.

2. Step 2: Differentiate from other monitoring types

   It is not about training data quality or training speed but about output behavior over time.

3. Final Answer:

   To track changes in the model's output predictions over time -> Option B

4. Quick Check:

   Prediction monitoring = track output changes [OK]

Hint: Focus on what is monitored: model outputs, not inputs or speed [OK]

Common Mistakes:

• Confusing prediction monitoring with data quality monitoring
• Thinking it speeds up training
• Assuming it increases dataset size

2. Which of the following is the correct way to calculate the distribution of predictions in Python using NumPy?

easy

A. np.sort(predictions, bins=10)

B. np.mean(predictions, bins=10)

C. np.sum(predictions, bins=10)

D. np.histogram(predictions, bins=10)

Solution

1. Step 1: Identify the function for distribution calculation

   NumPy's np.histogram calculates the frequency distribution of values in bins.

2. Step 2: Check other options

   np.mean calculates average, np.sum sums values, and np.sort sorts values, none calculate distribution.

3. Final Answer:

   np.histogram(predictions, bins=10) -> Option D

4. Quick Check:

   Distribution = histogram [OK]

Hint: Use np.histogram to get frequency counts in bins [OK]

Common Mistakes:

• Using mean or sum instead of histogram for distribution
• Trying to sort to get distribution
• Passing wrong arguments to functions

3. Given the following Python code snippet for monitoring prediction distribution, what will be the output?

import numpy as np
predictions = np.array([0.1, 0.4, 0.35, 0.8, 0.9])
hist, bins = np.histogram(predictions, bins=3)
print(hist)

medium

A. [3 1 1]

B. [1 2 2]

C. [2 1 2]

D. [2 2 1]

Solution

1. Step 1: Understand bin edges

   With bins=3, the range 0.1 to 0.9 is split into 3 equal parts: approx [0.1-0.4), [0.4-0.7), [0.7-1.0].

2. Step 2: Count predictions in each bin

   Bin 1: 0.1, 0.4 (0.4 is right edge, goes to next bin) -> 0.1 only -> 1 count Bin 2: 0.4, 0.35 -> 0.35 and 0.4 -> 2 counts Bin 3: 0.8, 0.9 -> 2 counts

3. Step 3: Correct bin counts

   Actually, np.histogram includes left edge, excludes right except last bin. So bins: [0.1,0.4), [0.4,0.7), [0.7,1.0] Values: 0.1 in bin1 0.35 in bin1 0.4 in bin2 0.8 in bin3 0.9 in bin3 Counts: bin1=2, bin2=1, bin3=2

4. Final Answer:

   [2 1 2] -> Option C

5. Quick Check:

   Histogram counts = [2,1,2] [OK]

Hint: Remember np.histogram includes left edge, excludes right edge except last bin [OK]

Common Mistakes:

• Miscounting values on bin edges
• Assuming bins include right edge
• Confusing bin counts order

4. You have this monitoring code snippet that throws an error:

import numpy as np
predictions = [0.2, 0.5, 0.7]
hist, bins = np.histogram(predictions, bins='five')
print(hist)

What is the cause of the error?

medium

A. The bins parameter must be an integer or sequence, not a string

B. The predictions list must be a NumPy array, not a list

C. The print statement syntax is incorrect

D. np.histogram does not accept more than 3 values

Solution

1. Step 1: Check bins parameter type

   np.histogram expects bins as an integer or a sequence of bin edges, not a string like 'five'.

2. Step 2: Verify other parts

   Predictions can be a list or array, print syntax is correct, and np.histogram accepts any length array.

3. Final Answer:

   The bins parameter must be an integer or sequence, not a string -> Option A

4. Quick Check:

   Bins must be int or list, not string [OK]

Hint: Bins must be number or list, never a string [OK]

Common Mistakes:

• Thinking list input causes error
• Blaming print syntax
• Assuming np.histogram limits input size

5. You want to detect if your model's prediction distribution has shifted significantly from the baseline. Which approach is best to implement in your monitoring pipeline?

hard

A. Calculate the KL divergence between baseline and current prediction distributions regularly

B. Only check if the average prediction value changes

C. Retrain the model every day regardless of prediction changes

D. Ignore distribution changes and focus on input data monitoring

Solution

1. Step 1: Understand distribution shift detection

   KL divergence measures how one distribution differs from another, ideal for detecting prediction shifts.

2. Step 2: Evaluate other options

   Checking only average misses distribution shape changes; retraining blindly wastes resources; ignoring prediction changes misses key signals.

3. Final Answer:

   Calculate the KL divergence between baseline and current prediction distributions regularly -> Option A

4. Quick Check:

   Use KL divergence for distribution shift detection [OK]

Hint: Use KL divergence to compare distributions, not just averages [OK]

Common Mistakes:

• Monitoring only average values
• Retraining without monitoring
• Ignoring prediction distribution shifts