Probability calibration in ML Python - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

What does probability calibration mean in machine learning?

Imagine you have a model that predicts the chance of rain tomorrow. What does it mean if the model is well calibrated?

A The model has the highest accuracy among all models.

B The model always predicts rain when it actually rains.

C The model predictions are always either 0 or 1.

D The predicted probabilities match the actual frequencies of rain over many days.

💡 Hint

Think about how predicted chances relate to what really happens over time.

Probability calibration means that when the model says 70% chance, it actually rains about 70% of those times. This matches predicted probabilities to real outcomes.

❓ Predict Output

intermediate

Output of calibration curve code snippet

What output does this code produce?

ML Python

from sklearn.calibration import calibration_curve
import numpy as np

# True labels
y_true = np.array([0, 0, 1, 1, 1, 0, 1, 0, 1, 0])
# Predicted probabilities
y_prob = np.array([0.1, 0.4, 0.35, 0.8, 0.9, 0.2, 0.7, 0.3, 0.85, 0.05])

prob_true, prob_pred = calibration_curve(y_true, y_prob, n_bins=3)
print(np.round(prob_true, 2))

A [0. 0.5 1. ]

B [0.33 0.67 0.75]

C [0.2 0.6 0.8]

D [0.25 0.67 1. ]

💡 Hint

Check how the predicted probabilities are grouped into bins and how many true positives are in each bin.

The calibration_curve function (default strategy='uniform') groups predictions into equal-width bins and computes the fraction of positives in each non-empty bin. Here, the fractions are 0.0 (low bin), 0.5 (mid bin), and 1.0 (high bin).

❓ Model Choice

advanced

Best calibration method for a binary classifier with overconfident predictions

You have a binary classifier whose predicted probabilities are often too close to 0 or 1, making it overconfident. Which calibration method is most suitable to fix this?

A Isotonic Regression (a non-parametric calibration method)

B Platt Scaling (logistic regression on the model's scores)

C No calibration needed, just retrain the model

D Use a decision tree classifier instead

💡 Hint

Consider which method can handle non-linear calibration well.

Isotonic Regression is a flexible, non-parametric method that can correct overconfident predictions by fitting a monotonic function, often better than Platt Scaling for such cases.

❓ Metrics

advanced

Which metric directly measures calibration quality?

Among these metrics, which one directly measures how well predicted probabilities match observed outcomes?

A Accuracy

B Brier Score

C Precision

D Recall

💡 Hint

Think about a metric that compares predicted probabilities to actual binary outcomes.

The Brier Score measures the mean squared difference between predicted probabilities and actual outcomes, directly assessing calibration quality.

🔧 Debug

expert

Why does this calibration code raise an error?

What error does this code raise and why?

ML Python

from sklearn.calibration import CalibratedClassifierCV
from sklearn.linear_model import LogisticRegression
import numpy as np

X = np.array([[1, 2], [3, 4], [5, 6], [7, 8]])
y = np.array([0, 1, 0, 1])

model = LogisticRegression()
calibrated = CalibratedClassifierCV(model, method='sigmoid')
calibrated.fit(X, y)

# Predict probabilities
probs = calibrated.predict_proba(X)

A ValueError: 'base_estimator' should be fitted before calibration

B NotFittedError: This CalibratedClassifierCV instance is not fitted yet

C No error, code runs successfully

D TypeError: method='sigmoid' is invalid

💡 Hint

Check the standard usage of CalibratedClassifierCV with an unfitted base estimator.

No error occurs. CalibratedClassifierCV clones the unfitted base estimator internally and fits copies of it on cross-validation folds during its own fit method.