Probability calibration in ML Python - Model Pipeline Trace

Model Pipeline - Probability calibration

This pipeline shows how a model's predicted probabilities are adjusted to better match true outcome frequencies. It improves trust in predictions by making probabilities more accurate.

Data Flow - 5 Stages

1Raw data input

1000 rows x 5 columns→Collect features and labels for classification→1000 rows x 5 columns

Features: age=30, income=50000, label=1

↓

2Train/test split

1000 rows x 5 columns→Split data into training and testing sets→800 rows x 5 columns (train), 200 rows x 5 columns (test)

Train: age=25, income=40000, label=0; Test: age=40, income=60000, label=1

↓

3Train base classifier

800 rows x 4 feature columns→Train model to predict class probabilities→Model with probability outputs

Model predicts 0.7 probability for class 1 on a sample

↓

4Predict probabilities on validation set

200 rows x 4 feature columns→Generate predicted probabilities→200 rows x 1 probability column

Predicted probability: 0.7 for class 1

↓

5Calibrate probabilities

200 rows x 1 probability column→Apply calibration method (e.g., Platt scaling or isotonic regression)→200 rows x 1 calibrated probability column

Raw probability 0.7 calibrated to 0.75

Training Trace - Epoch by Epoch

Loss
0.5 |****
0.4 |******
0.3 |********
0.2 |**********
     1 2 3 4 5 Epochs

Epoch	Loss ↓	Accuracy ↑	Observation
1	0.45	0.75	Initial training with moderate loss and accuracy
2	0.38	0.80	Loss decreased, accuracy improved
3	0.33	0.83	Continued improvement in loss and accuracy
4	0.30	0.85	Model converging with better predictions
5	0.28	0.86	Final epoch with stable loss and accuracy

Prediction Trace - 3 Layers

Layer 1: Base model prediction

Layer 2: Calibration function applied

Layer 3: Final calibrated output

Model Quiz - 3 Questions

Why do we calibrate predicted probabilities?

A To increase model accuracy only

B To make predicted probabilities match actual outcome frequencies

C To reduce the number of features

D To speed up training

Key Insight

Probability calibration improves the trustworthiness of model predictions by adjusting raw probabilities to better reflect true chances of outcomes. This helps users make better decisions based on model outputs.