Bird
Raised Fist0

Feature Engineering - Why Features Matter More Than Models

THEORYintermediateInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Feature Importance Mirage
The issue started immediately after deploying a new feature set last Friday.

A recommendation system for an e-commerce platform serving personalized product suggestions to 500k users daily.

Symptom
Model accuracy remains stable, but recall has dropped to 0, and AUC has decreased from 0.85 to 0.70 after a recent feature update.
📡
Signals
metric
Recall on the test set
Recall dropped from 0.75 to 0.00 after the feature update
metric
AUC on live traffic
AUC decreased from 0.85 to 0.70 post-deploy
feature_importance
Feature importance ranking
New feature 'user_purchase_history' ranked #1 with importance 0.30, overshadowing all other features
distribution
Distribution of 'user_purchase_history' feature
Highly skewed with 95% of values being zero
log
Deployment logs
No errors or warnings during deployment
🔬
Hypotheses
The new feature 'user_purchase_history' is causing the model to overfit to a small subset of users.
Supporting Signals
Feature importance rankingDistribution of 'user_purchase_history' feature
Verify: Check if the model's performance is disproportionately high for users with non-zero 'user_purchase_history'.
df[df['user_purchase_history'] > 0]['target'].mean() > df['target'].mean()
The new feature is not generalizing well across the user base.
Supporting Signals
Distribution of 'user_purchase_history' feature
Verify: Analyze the correlation between 'user_purchase_history' and target variable across different user segments.
df.groupby('user_segment').apply(lambda x: x[['user_purchase_history', 'target']].corr().iloc[0, 1])
🎯
Root Cause
The 'user_purchase_history' feature is highly skewed and dominates the model's decision-making, leading to poor generalization.
Why It Happened

The feature was added without proper normalization or consideration of its distribution, causing it to overshadow other features.

🔧
Fix Strategy
ImmediateRollback to the previous feature set to restore model performance.
Short-termNormalize the 'user_purchase_history' feature and retrain the model.
Long-termImplement feature scaling and regularization to prevent any single feature from dominating.
🐍
Python Fix
Python
from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler()
df['user_purchase_history'] = scaler.fit_transform(df[['user_purchase_history']])
✗
Common Wrong Diagnoses
▼ Show
The model architecture is flawed.
Why candidates think this: The drastic drop in recall suggests a fundamental model issue.
Why wrong: The model performed well before the feature update, indicating the architecture is sound.
Data leakage between training and validation sets.
Why candidates think this: Stable validation AUC might suggest leakage.
Why wrong: The issue is with feature dominance, not data leakage.
📊
Monitoring to Add
  • Monitor feature importance changes post-deployment
  • Track distribution shifts in key features
  • Set alerts for drastic drops in recall or AUC
📋
Process Change

Introduce a feature review process before deployment to assess potential impact on model performance.

🏗️
Architectural Change

Adopt a feature store that enforces distribution checks and normalization before features are used in production.

💡
Key Insight
Feature engineering can drastically alter model behavior, often more than model architecture changes.

Practice

(1/5)
1. 
import torch
import torch.nn as nn

conv = nn.Conv2d(in_channels=1, out_channels=1, kernel_size=3, stride=1, padding=0)
input_tensor = torch.ones((1, 1, 5, 5))
output_tensor = conv(input_tensor)
print(output_tensor.shape)
What is the output shape of the output_tensor?
easy
A. (1, 1, 5, 5)
B. (1, 1, 4, 4)
C. (1, 1, 2, 2)
D. (1, 1, 3, 3)

Solution

  1. Step 1: Calculate output dimensions

    Using the formula: ((W - F + 2P) / S) + 1, where W=5, F=3, P=0, S=1, the output size is ((5-3+0)/1)+1 = 3.

  2. Step 2: Eliminate wrong options

    (1, 1, 5, 5) is incorrect because the size doesn't remain the same. (1, 1, 4, 4) is incorrect because it assumes incorrect padding. (1, 1, 2, 2) is incorrect due to incorrect calculation.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Use output size formula for CNNs. [OK]
Hint: Use output size formula for CNNs. [OK]
Common Mistakes:
2. In the context of Random Forests, what is the primary purpose of using bagging?
easy
A. To increase the model's interpretability
B. To enhance the model's feature selection
C. To improve the model's computational efficiency
D. To reduce the model's variance

Solution

  1. Step 1: Understand bagging's role

    Bagging, or Bootstrap Aggregating, is used to reduce variance by averaging predictions from multiple models trained on different subsets of the data.

  2. Step 2: Eliminate wrong options

    To increase the model's interpretability is incorrect because bagging does not inherently increase interpretability. To enhance the model's feature selection is wrong as feature selection is not the primary goal of bagging. To improve the model's computational efficiency is incorrect because bagging typically increases computational cost due to multiple models.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Bagging reduces variance by averaging. [OK]
Hint: Bagging reduces variance by averaging. [OK]
Common Mistakes:
3. When choosing between LSTM and GRU for a time series prediction task with limited computational resources, which is generally more appropriate and why?
medium
A. LSTM, because it has more gates and thus more control over information flow.
B. LSTM, because it can handle longer sequences better than GRU.
C. GRU, because it is computationally less expensive with fewer parameters.
D. GRU, because it is always more accurate than LSTM.

Solution

  1. Step 1: Compare LSTM and GRU.

    GRU is simpler and often faster due to fewer parameters, making it suitable for limited resources.

  2. Step 2: Eliminate wrong options.

    LSTM's complexity isn't always needed, GRU isn't always more accurate, and LSTM's longer sequence handling isn't a universal advantage.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    GRU: fewer parameters, faster. [OK]
Hint: GRU: fewer parameters, faster. [OK]
Common Mistakes:
4. When monitoring an ML model in production, which approach is better for detecting data drift and why?
medium
A. Monitor statistical properties of input features.
B. Use a static threshold on model accuracy.
C. Rely on user feedback to identify issues.
D. Increase the frequency of model retraining.

Solution

  1. Step 1: Identify effective drift detection

    Monitoring statistical properties helps detect subtle changes in input data.

  2. Step 2: Eliminate wrong options

    Use a static threshold on model accuracy. is reactive, not proactive. Rely on user feedback to identify issues. is unreliable and subjective. Increase the frequency of model retraining. addresses performance decay, not drift detection.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Monitor input stats for drift detection [OK]
Hint: Monitor input stats for drift detection [OK]
Common Mistakes:
5. You have a model that is overfitting. Which approach is more suitable to address this issue?
medium
A. Add dropout layers to the model.
B. Reduce the size of the training dataset.
C. Use a smaller learning rate.
D. Increase the model complexity by adding more layers.

Solution

  1. Step 1: Identify methods to combat overfitting

    Dropout is a regularization technique that helps prevent overfitting by randomly dropping units during training.

  2. Step 2: Eliminate wrong options

    Increase the model complexity by adding more layers. increases overfitting risk, Reduce the size of the training dataset. reduces data, worsening generalization, and Use a smaller learning rate. affects convergence speed, not overfitting directly.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Dropout reduces overfitting [OK]
Hint: Dropout reduces overfitting [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard