Probability calibration in ML Python - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is probability calibration in machine learning?

Probability calibration is the process of adjusting a model's predicted probabilities so they better reflect the true likelihood of an event happening. For example, if a model says 70% chance of rain, it should actually rain about 70% of those times.

beginner

Why is probability calibration important?

It helps make predictions more trustworthy and useful. Well-calibrated probabilities allow better decision-making, like knowing when to trust a model's prediction or how to weigh risks.

intermediate

Name two common methods for probability calibration.

Two common methods are Platt Scaling, which fits a logistic regression on the model's scores, and Isotonic Regression, which fits a flexible non-decreasing function to adjust probabilities.

intermediate

What does a perfectly calibrated model's reliability diagram look like?

It is a diagonal line from bottom-left to top-right, meaning predicted probabilities match observed frequencies exactly. For example, predictions of 0.8 correspond to events happening 80% of the time.

intermediate

How can you check if a model's probabilities are well calibrated?

You can use calibration plots (reliability diagrams) or metrics like the Brier score, which measures the mean squared difference between predicted probabilities and actual outcomes.

What does probability calibration adjust in a model?

A The model's training data size

B The model's input features

C The predicted probabilities to better match true event frequencies

D The model's architecture

Probability calibration adjusts the predicted probabilities so they better reflect the true likelihood of events.

Which method fits a logistic regression to calibrate probabilities?

A Isotonic Regression

B Random Forest

C K-Means Clustering

D Platt Scaling

Platt Scaling fits a logistic regression model on the original model's scores to calibrate probabilities.

What shape does a reliability diagram have for a perfectly calibrated model?

A Diagonal line from bottom-left to top-right

B Vertical line

C Horizontal line

D Random scattered points

A diagonal line means predicted probabilities match observed frequencies exactly.

Which metric measures the mean squared difference between predicted probabilities and actual outcomes?

A Brier score

B Precision

C Recall

D Accuracy

The Brier score measures how close predicted probabilities are to actual outcomes.

Why might a model with high accuracy still need probability calibration?

A Because accuracy measures probability quality

B Because probabilities might not reflect true likelihoods

C Because calibration changes model architecture

D Because calibration increases training speed

High accuracy does not guarantee that predicted probabilities are well calibrated.

Explain in your own words what probability calibration is and why it matters.

Think about how confident a model's predictions should be to match reality.

Describe two methods used for probability calibration and how they differ.

One fits a logistic curve, the other fits a flexible shape.