Bird
Raised Fist0

Handling Missing Data - Imputation Strategies and When to Drop

THEORYintermediateInterview freq: high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Silent Recall Killer
The issue started after a new feature was added to the model last Monday.

Fraud detection model for a financial services company, processing 500K transactions daily.

Symptom
Model accuracy remains stable at 0.92, but recall has dropped to 0.0 for the last week.
📡
Signals
metric
Recall on the validation set is 0.85, but 0.0 on live data.
Recall dropped to 0.0 on live data after new feature deployment.
feature_importance
Feature importance ranking for 'transaction_location' changed significantly.
'transaction_location' importance dropped from rank #3 (0.15) to rank #20 (0.01) post-deploy.
distribution
Distribution of missing values in 'transaction_location' feature.
Missing values increased from 5% to 40% after new feature deployment.
log
Logs show increased warnings about missing data handling.
Frequent warnings: 'Imputation strategy not applied for 'transaction_location'.
metric
AUC on live data dropped from 0.91 to 0.75.
AUC dropped significantly after new feature deployment.
🔬
Hypotheses
The new feature introduced missing data that was not properly handled.
Supporting Signals
Recall on live data is 0.0Distribution of missing values increasedLogs show warnings about missing data handling
Verify: Check the preprocessing pipeline for missing data handling on the new feature.
df['transaction_location'].isnull().mean()
The model is overfitting to the validation set due to temporal leakage.
Supporting Signals
Recall on validation set is highAUC on live data dropped
Verify: Examine the temporal split strategy in the validation set.
df.groupby('date')['target'].mean().plot()
🎯
Root Cause
The new feature 'transaction_location' had a high percentage of missing values that were not imputed, leading to zero recall on live data.
Why It Happened

The deployment of the new feature did not include an updated imputation strategy for handling missing values.

🔧
Fix Strategy
ImmediateRollback to the previous model version without the new feature.
Short-termImplement a robust imputation strategy for 'transaction_location'.
Long-termIntegrate automated checks for missing data handling in the deployment pipeline.
🐍
Python Fix
Python
df['transaction_location'].fillna(df['transaction_location'].mode()[0], inplace=True)
✗
Common Wrong Diagnoses
▼ Show
The model is overfitting to the validation set.
Why candidates think this: High recall on validation but low on live data suggests overfitting.
Why wrong: The issue is not overfitting but missing data in live data not present in validation.
The new feature is irrelevant and should be removed.
Why candidates think this: Drop in feature importance suggests irrelevance.
Why wrong: The drop is due to missing data, not feature irrelevance.
📊
Monitoring to Add
  • Monitor percentage of missing values for all features post-deployment.
  • Set up alerts for significant drops in recall on live data.
  • Track feature importance changes after each deployment.
📋
Process Change

Require a checklist for data preprocessing steps, including missing data handling, before feature deployment.

🏗️
Architectural Change

Implement a data validation layer that checks for missing data and applies imputation before model inference.

💡
Key Insight
Missing data can silently cripple model performance if not properly handled, especially when new features are introduced.

Practice

(1/5)
1. In the context of object detection, which of the following is a key advantage of YOLO over R-CNN?
easy
A. Faster inference time
B. Higher accuracy on small objects
C. Better handling of occlusions
D. More robust to varying lighting conditions

Solution

  1. Step 1: Identify YOLO's strengths

    YOLO is known for its real-time processing capabilities due to its single-stage architecture.

  2. Step 2: Eliminate wrong options

    Higher accuracy on small objects: R-CNN is generally better for small objects. Better handling of occlusions: Handling occlusions is not a specific strength of YOLO. More robust to varying lighting conditions: Lighting conditions are not specifically addressed by YOLO.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    YOLO = Speed, R-CNN = Precision [OK]
Hint: YOLO = Speed, R-CNN = Precision [OK]
Common Mistakes:
2. In the context of diagnosing overfitting, which of the following is a clear indicator that a model is overfitting?
easy
A. The model performs well on both training and validation datasets.
B. The model performs well on the training dataset but poorly on the validation dataset.
C. The model performs poorly on both training and validation datasets.
D. The model's performance improves with more training data.

Solution

  1. Step 1: Identify overfitting symptoms

    Overfitting occurs when a model learns the training data too well, including noise, leading to poor generalization on unseen data.

  2. Step 2: Eliminate wrong options

    The model performs well on both training and validation datasets. indicates good generalization, The model performs poorly on both training and validation datasets. indicates underfitting, and The model's performance improves with more training data. suggests improvement, not overfitting.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Good train, bad validation = overfitting [OK]
Hint: Good train, bad validation = overfitting [OK]
Common Mistakes:
3. Given the following Python code snippet, what will be the output when calculating the Euclidean distance between two high-dimensional points?
import numpy as np
point1 = np.array([1, 2, 3, ..., 1000])
point2 = np.array([2, 3, 4, ..., 1001])
distance = np.linalg.norm(point1 - point2)
print(distance)
easy
A. 1.0
B. 1000.0
C. 0.0
D. 31.62

Solution

  1. Step 1: Calculate the difference between points.

    The difference is [1, 1, 1, ..., 1].

  2. Step 2: Calculate the Euclidean distance.

    The distance is sqrt(1000) = 31.62.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Euclidean distance in high dimensions is sqrt(n). [OK]
Hint: Euclidean distance in high dimensions is sqrt(n). [OK]
Common Mistakes:
4. A neural network model is not improving beyond a certain accuracy. What is the most likely cause?
medium
A. The model has reached its capacity
B. The model is underfitting
C. The learning rate is too high
D. The dataset is too small

Solution

  1. Step 1: Analyze the symptom

    If a model's accuracy plateaus, it often indicates that the model has reached its capacity to learn from the data.

  2. Step 2: Eliminate wrong options

    Options A and B would show different symptoms like oscillating loss or high training error. The dataset is too small would likely cause overfitting, not a plateau.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Plateau = model capacity limit [OK]
Hint: Plateau = model capacity limit [OK]
Common Mistakes:
5. A model trained with Adam optimizer is not converging. What is the most likely cause?
medium
A. Batch size is too large
B. Learning rate is too high
C. Adam is not suitable for non-convex problems
D. Weight decay is too low

Solution

  1. Step 1: Identify common issues with Adam.

    Adam can be sensitive to high learning rates, causing divergence.

  2. Step 2: Eliminate less likely causes.

    Adam handles non-convex problems well, and batch size/weight decay are less likely to cause non-convergence.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    High lr can cause Adam to diverge. [check]
Hint: High lr can cause Adam to diverge. [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard