Bird
Raised Fist0

Calibration - Why Accuracy Isn't Enough for Probability Outputs

THEORYadvancedInterview freq: high
Choose your preparation mode6 modes available
▶
Execution Trace
Step 1/4
Model Output Probabilities

Initial probabilities are 0.1, 0.4, and 0.8 for three classes.

🎯 Understanding initial model confidence levels is crucial for calibration.
model_probs = [0.1, 0.4, 0.8]
Calibration Function

Calibration function adjusts probabilities to better match true likelihoods.

🎯 Calibration functions like Platt scaling or isotonic regression are used here.
from sklearn.calibration import CalibratedClassifierCV
calibrated_model = CalibratedClassifierCV(base_estimator=model, method='sigmoid')
Calibrated Probabilities

Probabilities are adjusted to 0.05, 0.35, and 0.75.

🎯 Calibrated probabilities provide a more accurate reflection of true class likelihoods.
calibrated_probs = calibrated_model.predict_proba(X_test)
Comparison

Calibrated probabilities are closer to actual class frequencies.

🎯 Comparing calibrated vs. uncalibrated probabilities highlights calibration benefits.
print('Uncalibrated:', model_probs)
print('Calibrated:', calibrated_probs)
📊
Diagram
Diagram
┌──────────────────────────────────────────────────────────────┐
│                      Probability Calibration                 │
├──────────────────────────────────────────────────────────────┤
│                                                              │
│  Model Output Probabilities                                  │
│  ┌───────────────┐                                           │
│  │ 0.1  0.4  0.8 │                                           │
│  └───────────────┘                                           │
│         │                                                    │
│         ▼                                                    │
│  Calibration Function                                        │
│  ┌────────────────────────────────────────────────────────┐  │
│  │ Adjusts probabilities to better reflect true likelihood │  │
│  └────────────────────────────────────────────────────────┘  │
│         │                                                    │
│         ▼                                                    │
│  Calibrated Probabilities                                    │
│  ┌───────────────┐                                           │
│  │ 0.05 0.35 0.75│                                           │
│  └───────────────┘                                           │
│                                                              │
└──────────────────────────────────────────────────────────────┘
  • Model Output Probabilities: Initial probabilities from the model
  • Calibration Function: Adjusts probabilities to improve accuracy
  • Calibrated Probabilities: Adjusted probabilities reflecting true likelihood
🔑
Key Moments
▲
Q: Why is calibration necessary for model probabilities?
A: Calibration ensures that the predicted probabilities reflect the true likelihood of outcomes, improving decision-making.
Q: How does the calibration function adjust probabilities?
A: It uses techniques like Platt scaling to adjust the model's output probabilities to better match observed frequencies.
Q: What is the impact of using uncalibrated probabilities?
A: Uncalibrated probabilities can lead to overconfident or underconfident predictions, affecting decision quality.

This diagram and trace explain the importance of calibrating model output probabilities. Initially, models may output probabilities that do not accurately reflect true likelihoods. Calibration functions, such as Platt scaling, adjust these probabilities to better match observed data. This process results in calibrated probabilities that provide a more reliable basis for decision-making. Understanding this concept is crucial for improving model performance in real-world applications where probability estimates guide critical decisions.

Primary Focus

✓ Understanding the importance of calibration in probability outputs

Practice

(1/5)
1.
import numpy as np
from sklearn.preprocessing import StandardScaler

X = np.array([[1, 2], [3, 4], [5, 6]])
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
print(X_scaled.mean(axis=0))
What does this code output?
easy
A. [1. 1.]
B. [0.5 0.5]
C. [0. 0.]
D. [0. 1.]

Solution

  1. Step 1: Understand standardization.

    StandardScaler standardizes features by removing the mean and scaling to unit variance.
  2. Step 2: Calculate the mean of standardized data.

    After standardization, the mean of each feature should be 0.
  3. Final Answer:

    Option C -> Option C
  4. Quick Check:

    StandardScaler centers data to mean 0. [OK]
Hint: StandardScaler centers data to mean 0. [OK]
Common Mistakes:
2. Given the following Python code snippet, what will be the output?
import numpy as np
business_goals = np.array(['increase sales', 'reduce churn', 'improve satisfaction'])
ml_problems = np.array(['classification', 'regression', 'clustering'])
result = np.char.add(business_goals, ml_problems)
print(result)
easy
A. ['increase salesclassification', 'reduce churnregression', 'improve satisfactionclustering']
B. ['increase sales', 'reduce churn', 'improve satisfaction']
C. ['classification', 'regression', 'clustering']
D. Throws an error due to incompatible types

Solution

  1. Step 1: Understand the operation

    The np.char.add function concatenates corresponding elements of two arrays element-wise.
  2. Step 2: Eliminate wrong options

    ['increase sales', 'reduce churn', 'improve satisfaction'] and C are incorrect because they ignore the concatenation operation. Throws an error due to incompatible types is incorrect because np.char.add handles string arrays correctly.
  3. Final Answer:

    Option A -> Option A
  4. Quick Check:

    np.char.add concatenates strings element-wise. [OK]
Hint: np.char.add concatenates strings element-wise. [OK]
Common Mistakes:
3. You have a dataset with a large number of features and limited computational resources. Which feature selection method is most appropriate?
medium
A. Wrapper method
B. Embedded method
C. Recursive feature elimination
D. Filter method

Solution

  1. Step 1: Consider computational efficiency.

    Filter methods are computationally inexpensive as they do not involve training models.
  2. Step 2: Eliminate wrong options.

    Wrapper methods and recursive feature elimination are computationally intensive. Embedded methods require model training, which can be resource-heavy.
  3. Final Answer:

    Option D -> Option D
  4. Quick Check:

    Filter = low cost, no model. [OK]
Hint: Filter methods are computationally cheap. [OK]
Common Mistakes:
4. When should you avoid using Naive Bayes despite its simplicity and speed?
medium
A. When the dataset is very large.
B. When the dataset has missing values.
C. When you need a probabilistic output.
D. When features are highly correlated.

Solution

  1. Step 1: Identify Naive Bayes Limitations

    Naive Bayes assumes feature independence given the class label.
  2. Step 2: Evaluate Each Option

    When the dataset is very large. is incorrect because Naive Bayes handles large datasets well. When you need a probabilistic output. is incorrect because Naive Bayes provides probabilistic outputs. When the dataset has missing values. is incorrect because Naive Bayes can handle missing values with some preprocessing.
  3. Final Answer:

    Option D -> Option D
  4. Quick Check:

    Correlated features break Naive Bayes. [[check]]
Hint: Correlated features break Naive Bayes. [[check]]
Common Mistakes:
5. How would you adapt dropout for a recurrent neural network (RNN) to prevent overfitting without disrupting temporal dependencies?
hard
A. Apply dropout to the input and output layers only.
B. Increase the dropout rate progressively over time.
C. Apply dropout to the hidden states at each time step.
D. Use the same dropout mask at each time step.

Solution

  1. Step 1: Understand dropout in RNNs

    Using the same dropout mask at each time step helps maintain temporal dependencies.
  2. Step 2: Eliminate wrong options

    Apply dropout to the input and output layers only. is incorrect because it doesn't address hidden state dropout. Apply dropout to the hidden states at each time step. is incorrect because applying dropout to hidden states at each time step can disrupt dependencies. Increase the dropout rate progressively over time. is incorrect because increasing dropout rate over time is not a standard practice and can lead to underfitting.
  3. Final Answer:

    Option D -> Option D
  4. Quick Check:

    Same mask preserves RNN dependencies. [OK]
Hint: Same mask preserves RNN dependencies. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard