Bird
Raised Fist0

Statistical Significance in ML - P-Values, Confidence Intervals, Effect Size

THEORYintermediateInterview freq: medium
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Silent Recall Killer
Issue started after the latest model update last Monday.

A fraud detection model for a financial services company, processing 500K transactions daily.

Symptom
Model accuracy remains stable at 0.95, but recall has dropped to 0.0.
📡
Signals
metric
Recall on the validation set
Recall dropped from 0.85 to 0.0 after the update
metric
Precision on the validation set
Precision remains stable at 0.92
distribution
Distribution of predicted probabilities
Shifted towards 0.0, with most predictions below 0.1
feature_importance
Feature importance ranking
Feature 'transaction_amount' importance dropped from 0.25 to 0.05
log
Model training logs
No significant changes in training loss or validation loss
🔬
Hypotheses
The model is overfitting to a specific feature that has changed distribution.
Supporting Signals
feature_importancedistribution
Verify: Check the distribution of 'transaction_amount' in the training and validation datasets.
import matplotlib.pyplot as plt; plt.hist(training_df['transaction_amount'], bins=50, alpha=0.5, label='Training'); plt.hist(validation_df['transaction_amount'], bins=50, alpha=0.5, label='Validation'); plt.legend(); plt.show()
The threshold for classification is set too high, causing all predictions to be negative.
Supporting Signals
distribution
Verify: Evaluate the model's performance at different thresholds.
import numpy as np; from sklearn.metrics import recall_score; thresholds = np.arange(0.0, 1.0, 0.1); recalls = [recall_score(y_true, y_pred > t) for t in thresholds]; plt.plot(thresholds, recalls); plt.show()
🎯
Root Cause
The classification threshold was inadvertently set too high during the last update.
Why It Happened

A configuration error during deployment set the threshold to 0.9 instead of the intended 0.5.

🔧
Fix Strategy
ImmediateRollback to the previous model version with the correct threshold.
Short-termCorrect the threshold configuration in the current deployment.
Long-termImplement a dynamic threshold selection based on validation set performance.
🐍
Python Fix
Python
threshold = 0.5; y_pred = (model.predict_proba(X)[:, 1] > threshold).astype(int)
✗
Common Wrong Diagnoses
▼ Show
The model is overfitting to the training data.
Why candidates think this: The drop in recall suggests poor generalization.
Why wrong: The precision remains stable, indicating the model is not overfitting.
Feature 'transaction_amount' is no longer relevant.
Why candidates think this: The drop in feature importance suggests irrelevance.
Why wrong: The issue is with the threshold, not the feature's relevance.
📊
Monitoring to Add
  • Monitor recall and precision metrics post-deployment.
  • Set up alerts for significant shifts in predicted probability distributions.
  • Track feature importance changes across deployments.
📋
Process Change

Introduce a checklist for deployment configurations, including threshold settings.

🏗️
Architectural Change

Develop a CI/CD pipeline that includes automated threshold tuning based on validation metrics.

💡
Key Insight
A misconfigured threshold can silently kill recall without affecting other metrics.

Practice

(1/5)
1. A model trained with K-Fold cross-validation is underperforming on unseen data. What is a likely cause?
medium
A. The model is overfitting due to too many folds.
B. Data leakage between training and validation sets.
C. The model is underfitting due to too few folds.
D. The model is not complex enough for the data.

Solution

  1. Step 1: Consider data leakage as a common issue.

    Data leakage can occur if the same data points appear in both training and validation sets.

  2. Step 2: Eliminate wrong options.

    Too many folds (A) typically doesn't cause overfitting, too few folds (C) would not cause underperformance on unseen data, model complexity (D) is unrelated to cross-validation setup.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Check for data leakage in CV. [OK]
Hint: Check for data leakage in CV. [OK]
Common Mistakes:
2. A model trained on a dataset is performing exceptionally well on the test set but poorly on new unseen data. What is the most likely cause?
medium
A. The model is overfitting due to too many parameters.
B. The test set contains data leakage from the training set.
C. The model has not been trained for enough epochs.
D. The feature scaling was not applied correctly.

Solution

  1. Step 1: Analyze the symptoms

    High test performance but poor real-world performance suggests leakage.

  2. Step 2: Eliminate other causes

    The model is overfitting due to too many parameters. would affect both test and real-world data, The model has not been trained for enough epochs. would not cause high test performance, The feature scaling was not applied correctly. would affect all data equally.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    High test, low real-world = leakage [OK]
Hint: High test, low real-world = leakage [OK]
Common Mistakes:
3. A KNN model is performing poorly on a high-dimensional dataset. What is the most likely cause?
medium
A. The model has too few neighbors.
B. The distance metric is ineffective in high dimensions.
C. The dataset is too small.
D. The model is overfitting due to too many neighbors.

Solution

  1. Step 1: Consider high-dimensional effects

    In high dimensions, distance metrics often become less effective.

  2. Step 2: Eliminate wrong options

    The model has too few neighbors. is incorrect as increasing neighbors may not help. The dataset is too small. is wrong because size isn't the main issue. The model is overfitting due to too many neighbors. is incorrect as overfitting is less likely with many neighbors.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    High dimensions dilute distance metrics. [OK]
Hint: High dimensions dilute distance metrics. [OK]
Common Mistakes:
4. A Random Forest model is underperforming on out-of-distribution (OOD) data. What is the most likely cause?
medium
A. The model has too many trees
B. The training data lacks diversity
C. The model is overfitting to the training data
D. The model uses too few features per split

Solution

  1. Step 1: Identify OOD challenges

    OOD performance issues often arise from a lack of diversity in training data.

  2. Step 2: Analyze options

    The training data lacks diversity is correct because diverse training data is crucial for generalization. The model has too many trees is incorrect as more trees generally improve robustness. The model is overfitting to the training data is incorrect because overfitting affects in-distribution performance more. The model uses too few features per split is incorrect as using fewer features can help generalization.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    OOD issues often stem from non-diverse data. [OK]
Hint: OOD issues often stem from non-diverse data. [OK]
Common Mistakes:
5. When choosing between BPE and WordPiece for a language model, which scenario makes WordPiece a better choice?
medium
A. When the vocabulary size needs to be minimized.
B. When handling a language with a high degree of inflection.
C. When the model needs to handle rare words efficiently.
D. When the model is primarily used for text generation tasks.

Solution

  1. Step 1: Identify the strengths of WordPiece.

    WordPiece is designed to handle rare words efficiently by breaking them into subwords.

  2. Step 2: Eliminate wrong options.

    Minimizing vocabulary size (A) and handling inflection (B) are not specific strengths of WordPiece; text generation (D) is not a specific advantage.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    WordPiece excels with rare words. [OK]
Hint: WordPiece excels with rare words. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard