Bird
Raised Fist0

Data Leakage - The Silent Killer of ML Models

THEORYadvancedInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Mysterious Recall Vanishing
Issue started after the last model update two weeks ago

Fraud detection model for a financial service processing 500K transactions daily

Symptom
Model accuracy remains stable at 0.95, but recall has dropped to 0
📡
Signals
metric
Recall on live data
Recall dropped from 0.85 to 0 after the last update
metric
Validation AUC
Validation AUC remains stable at 0.92
feature_importance
Feature importance ranking
Feature 'transaction_time' importance increased from 0.05 to 0.25
distribution
Distribution of 'transaction_time' feature
Distribution shifted significantly post-update
log
Model training logs
Logs show 'transaction_time' was used in both training and validation
🔬
Hypotheses
Data leakage from 'transaction_time' feature
Supporting Signals
Feature 'transaction_time' importance increasedLogs show 'transaction_time' was used in both training and validation
Verify: Check if 'transaction_time' is leaking future information into training
print(df[['transaction_time', 'target']].head())
Temporal shift in 'transaction_time' feature
Supporting Signals
Distribution of 'transaction_time' feature shifted
Verify: Compare pre and post-update distributions of 'transaction_time'
df['transaction_time'].hist(bins=50, alpha=0.5, label='Pre-update'); df['transaction_time'].hist(bins=50, alpha=0.5, label='Post-update')
🎯
Root Cause
Data leakage due to 'transaction_time' feature being used improperly
Why It Happened

A recent change in the data pipeline allowed 'transaction_time' to be included in both training and validation sets without proper handling

🔧
Fix Strategy
ImmediateRollback to the previous model version
Short-termRemove 'transaction_time' from the training and validation datasets
Long-termImplement a temporal holdout strategy to prevent future leakage
🐍
Python Fix
Python
df = df.drop(columns=['transaction_time'])
✗
Common Wrong Diagnoses
▼ Show
Model overfitting to training data
Why candidates think this: Stable validation AUC suggests overfitting might be the issue
Why wrong: Overfitting would affect both validation and live metrics, but only live recall is affected
Feature engineering error
Why candidates think this: Feature importance change suggests a feature issue
Why wrong: The issue is not with feature engineering but with data leakage
📊
Monitoring to Add
  • Monitor feature importance changes post-deployment
  • Track recall and precision metrics separately
  • Set up alerts for significant shifts in feature distributions
  • Regularly audit data pipeline for potential leakage points
📋
Process Change

Introduce a mandatory review of feature inclusion criteria before model updates

🏗️
Architectural Change

Design the data pipeline to automatically exclude temporal features from validation

💡
Key Insight
Data leakage can silently degrade model performance, especially in metrics like recall, without affecting validation metrics.

Practice

(1/5)
1. A Random Forest model is underperforming on out-of-distribution (OOD) data. What is the most likely cause?
medium
A. The model has too many trees
B. The training data lacks diversity
C. The model is overfitting to the training data
D. The model uses too few features per split

Solution

  1. Step 1: Identify OOD challenges

    OOD performance issues often arise from a lack of diversity in training data.

  2. Step 2: Analyze options

    The training data lacks diversity is correct because diverse training data is crucial for generalization. The model has too many trees is incorrect as more trees generally improve robustness. The model is overfitting to the training data is incorrect because overfitting affects in-distribution performance more. The model uses too few features per split is incorrect as using fewer features can help generalization.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    OOD issues often stem from non-diverse data. [OK]
Hint: OOD issues often stem from non-diverse data. [OK]
Common Mistakes:
2. When considering bias mitigation strategies, which approach is more suitable for a scenario where you cannot alter the training data but need to ensure fairness in predictions?
medium
A. Pre-processing techniques
B. In-processing techniques
C. Data augmentation
D. Post-processing techniques

Solution

  1. Step 1: Identify the constraint.

    Cannot alter training data, need fairness in predictions.

  2. Step 2: Choose the suitable technique.

    Post-processing techniques adjust predictions to ensure fairness without altering the model or data.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Post-processing adjusts predictions, not data. [OK]
Hint: Post-processing adjusts predictions, not data. [OK]
Common Mistakes:
3. You implemented stratified sampling but notice that the sample still has class imbalance. What is the most likely cause?
medium
A. Data leakage from training set.
B. Using random sampling instead.
C. Incorrectly specified class proportions.
D. Insufficient sample size.

Solution

  1. Step 1: Identify the stratified sampling requirement.

    Stratified sampling requires correct class proportion specification.

  2. Step 2: Eliminate unrelated issues.

    Random Sampling (B) would not be used if stratified was intended. Data leakage (C) affects model performance, not sampling. Sample size (D) affects variability, not imbalance directly.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Check class proportions in stratified sampling [[check]]
Hint: Check class proportions in stratified sampling [[check]]
Common Mistakes:
4. How would you adapt KNN for a real-time recommendation system with millions of users and items?
hard
A. Implement approximate nearest neighbors with locality-sensitive hashing.
B. Use exact KNN with a large number of neighbors.
C. Reduce the number of features to improve speed.
D. Use a single global model for all users.

Solution

  1. Step 1: Consider scalability and speed

    Approximate methods like locality-sensitive hashing are suitable for large-scale, real-time systems.

  2. Step 2: Eliminate wrong options

    Use exact KNN with a large number of neighbors. is incorrect due to inefficiency. Reduce the number of features to improve speed. is wrong as feature reduction alone may not suffice. Use a single global model for all users. is incorrect because personalization is key in recommendations.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Approximate methods for real-time scale. [OK]
Hint: Approximate methods for real-time scale. [OK]
Common Mistakes:
5. You are tasked with improving a model's performance on a highly imbalanced dataset. Which approach is most appropriate to prevent overfitting while addressing class imbalance?
hard
A. Downsample the majority class and increase the learning rate.
B. Use SMOTE to oversample the minority class and increase model complexity.
C. Apply class weights in the loss function and use early stopping.
D. Use a simpler model with no regularization.

Solution

  1. Step 1: Address class imbalance and overfitting

    Class weights help balance the dataset, and early stopping prevents overfitting by halting training when validation performance stops improving.

  2. Step 2: Eliminate wrong options

    Use SMOTE to oversample the minority class and increase model complexity. increases complexity, risking overfitting, Downsample the majority class and increase the learning rate. may lead to instability, and Use a simpler model with no regularization. lacks regularization, increasing overfitting risk.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Class weights + early stopping = balanced, robust model [OK]
Hint: Class weights + early stopping = balanced, robust model [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard