Bird
Raised Fist0

Logistic Regression - Decision Boundary, Log-Odds, Calibration

THEORYbeginnerInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Misleading Calibration
Issue started after the last feature update two weeks ago.

A logistic regression model predicting customer churn for a telecom company, serving predictions for 500k customers monthly.

Symptom
Model accuracy is stable at 0.85, but recall has dropped to 0.0 for the positive class.
📡
Signals
metric
Recall for the positive class in the test set
Recall dropped from 0.75 to 0.0 after the feature update
feature_importance
Feature importance ranking in the model
New feature 'recent_plan_change' has the highest importance at 0.25
distribution
Distribution of predicted probabilities
Predicted probabilities are heavily skewed towards 0.0
log
Model training logs
No errors or warnings during training
metric
AUC score on validation set
AUC remains stable at 0.89
🔬
Hypotheses
The decision boundary is incorrectly set due to skewed log-odds.
Supporting Signals
Recall dropped to 0.0Predicted probabilities skewed towards 0.0
Verify: Check the distribution of log-odds and decision boundary threshold.
import numpy as np; import matplotlib.pyplot as plt; log_odds = np.log(model.predict_proba(X_test)[:, 1] / (1 - model.predict_proba(X_test)[:, 1])); plt.hist(log_odds, bins=50); plt.show()
The new feature 'recent_plan_change' is dominating the model's decision process.
Supporting Signals
Feature 'recent_plan_change' has highest importance
Verify: Evaluate model performance without the new feature.
X_test_reduced = X_test.drop(columns=['recent_plan_change']); X_train_reduced = X_train.drop(columns=['recent_plan_change']); model_reduced = LogisticRegression().fit(X_train_reduced, y_train); recall_reduced = recall_score(y_test, model_reduced.predict(X_test_reduced)); print(recall_reduced)
🎯
Root Cause
The decision boundary was miscalibrated due to the new feature skewing log-odds.
Why It Happened

The new feature 'recent_plan_change' introduced a strong bias, affecting the calibration of predicted probabilities.

🔧
Fix Strategy
ImmediateRollback to the previous model version without the new feature.
Short-termRecalibrate the decision boundary using a validation set that includes the new feature.
Long-termImplement a calibration layer to adjust predicted probabilities post-training.
🐍
Python Fix
Python
from sklearn.calibration import CalibratedClassifierCV; calibrated_model = CalibratedClassifierCV(base_estimator=model, method='sigmoid'); calibrated_model.fit(X_train, y_train);
✗
Common Wrong Diagnoses
▼ Show
The model is overfitting due to the new feature.
Why candidates think this: High feature importance suggests overfitting.
Why wrong: Overfitting would typically affect both accuracy and recall, not just recall.
The data pipeline introduced noise, affecting predictions.
Why candidates think this: Feature update coincides with recall drop.
Why wrong: Noise would likely degrade overall accuracy, not just recall.
📊
Monitoring to Add
  • Monitor recall for each class separately post-deployment.
  • Track distribution of predicted probabilities for skewness.
  • Set alerts for significant changes in feature importance rankings.
📋
Process Change

Include calibration checks in the model validation pipeline before deployment.

🏗️
Architectural Change

Incorporate a calibration step in the model training process to adjust decision boundaries dynamically.

💡
Key Insight
Calibration issues can silently degrade class-specific metrics even when overall metrics appear stable.

Practice

(1/5)
1. What will be the output of the following Python code snippet using NumPy for a simple linear regression prediction?
import numpy as np
X = np.array([[1, 2], [3, 4]])
weights = np.array([0.5, 0.5])
predictions = np.dot(X, weights)
print(predictions)
easy
A. [1.0, 2.0]
B. [2.5, 3.5]
C. [1.5, 3.5]
D. [2.0, 4.0]

Solution

  1. Step 1: Calculate the dot product

    The dot product of X and weights is calculated as follows: [1*0.5 + 2*0.5, 3*0.5 + 4*0.5] = [2.0, 4.0].

  2. Step 2: Eliminate wrong options

    Options A, B, and D result from incorrect calculations of the dot product.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Dot product: sum of element-wise products. [OK]
Hint: Dot product: sum of element-wise products. [OK]
Common Mistakes:
2. 
import shap
import xgboost

# Load data
X, y = shap.datasets.boston()

# Train model
model = xgboost.XGBRegressor().fit(X, y)

# Explain the model's predictions using SHAP
explainer = shap.Explainer(model, X)
shap_values = explainer(X)

# What is the shape of shap_values?
easy
A. (13, 506)
B. (506,)
C. (506, 13)
D. (13,)

Solution

  1. Step 1: Understand SHAP output.

    SHAP values are calculated for each feature of each instance.

  2. Step 2: Determine the shape based on input data.

    The dataset has 506 instances and 13 features, so SHAP values will have the shape (506, 13).

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    SHAP values match data shape: (instances, features). [OK]
Hint: SHAP values match data shape: (instances, features). [OK]
Common Mistakes:
3. Which regularization technique is most suitable for feature selection in a linear regression model?
easy
A. L1 Lasso
B. Elastic Net
C. L2 Ridge
D. Dropout

Solution

  1. Step 1: Identify the purpose of feature selection.

    L1 Lasso regularization can shrink some coefficients to zero, effectively selecting features.

  2. Step 2: Eliminate wrong options.

    L2 Ridge does not shrink coefficients to zero. Elastic Net combines L1 and L2 but is not as direct for feature selection. Dropout is not used in linear regression.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    L1 Lasso zeros coefficients for selection. [OK]
Hint: L1 Lasso zeros coefficients for selection. [OK]
Common Mistakes:
4. 
import numpy as np

x = np.array([1, 2, 3])
y = np.array([4, 5, 6])

result = np.dot(x, y)
print(result)
What does this code output?
easy
A. Error: shapes not aligned
B. [4, 10, 18]
C. 32
D. [5, 7, 9]

Solution

  1. Step 1: Understand the dot product operation.

    The dot product of two vectors is the sum of the products of their corresponding elements.

  2. Step 2: Calculate the dot product.

    (1*4) + (2*5) + (3*6) = 32.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Dot product sums element-wise products. [OK]
Hint: Dot product sums element-wise products. [OK]
Common Mistakes:
5. A model using wrapper methods for feature selection is overfitting. What is the most likely cause?
medium
A. Data leakage during feature selection
B. Inappropriate scoring function
C. Too few features selected
D. High variance in the dataset

Solution

  1. Step 1: Identify overfitting causes.

    Data leakage during feature selection can lead to overfitting as the model sees information from the validation set.

  2. Step 2: Eliminate wrong options.

    Too few features usually lead to underfitting. An inappropriate scoring function affects feature ranking, not directly overfitting. High variance can cause overfitting but is not specific to wrapper methods.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Data leakage = overfitting risk. [OK]
Hint: Data leakage causes overfitting. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard