Bird
Raised Fist0

Feature Selection - Filter, Wrapper, Embedded Methods

THEORYintermediateInterview freq: medium
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Feature Selection Fiasco
Started immediately after the last feature update, ongoing for 3 days.

A customer churn prediction model for a telecom company, processing 500k records daily.

Symptom
Model accuracy is stable at 0.85, but recall has dropped to 0.0 after the last feature update.
📡
Signals
metric
Recall on the test set
Recall dropped from 0.72 to 0.0 after the feature update
feature_importance
Feature importance ranking from the model
Feature 'recent_calls' importance dropped from 0.25 to 0.01
distribution
Distribution of the 'recent_calls' feature
Became skewed with 90% zeros after the update
log
Model training logs
No errors, but warnings about feature correlation detected
metric
AUC on the validation set
AUC remained stable at 0.88
🔬
Hypotheses
The new feature selection method removed critical features.
Supporting Signals
Recall dropped to 0.0Feature 'recent_calls' importance dropped
Verify: Check which features were removed by the new selection method.
selected_features = [f for f in all_features if f not in removed_features]; print(selected_features)
The 'recent_calls' feature distribution change affected model performance.
Supporting Signals
Distribution of 'recent_calls' became skewed
Verify: Analyze the impact of 'recent_calls' on model predictions.
df['recent_calls'].hist(); plt.show()
🎯
Root Cause
The new feature selection method removed the 'recent_calls' feature, which was critical for recall.
Why It Happened

A wrapper method was used that prioritized accuracy over recall, leading to the exclusion of features important for recall.

🔧
Fix Strategy
ImmediateRollback to the previous feature set to restore recall.
Short-termRe-evaluate the feature selection criteria to include features critical for recall.
Long-termImplement a hybrid feature selection approach that balances accuracy and recall.
🐍
Python Fix
Python
selected_features = previous_feature_set; model.fit(X_train[selected_features], y_train)
✗
Common Wrong Diagnoses
▼ Show
Assuming the model is overfitting due to stable AUC.
Why candidates think this: Stable AUC suggests the model is not generalizing well.
Why wrong: Overfitting would affect both AUC and recall, but only recall dropped.
Believing the issue is due to data leakage.
Why candidates think this: Data leakage can cause unexpected metric drops.
Why wrong: Data leakage would typically affect all metrics, not just recall.
📊
Monitoring to Add
  • Monitor recall on a daily basis post-deployment.
  • Track feature importance changes after each deployment.
  • Set alerts for drastic changes in feature distributions.
📋
Process Change

Incorporate recall as a key metric in feature selection evaluation.

🏗️
Architectural Change

Adopt a feature selection pipeline that allows for dynamic adjustment based on multiple metrics.

💡
Key Insight
Feature selection methods can inadvertently prioritize certain metrics over others, leading to critical performance drops in overlooked areas.

Practice

(1/5)
1. You are dealing with a dataset where both covariate and concept drift are suspected. Which approach is more suitable for handling this scenario?
medium
A. Use a static model trained on historical data.
B. Implement a model retraining strategy with periodic updates.
C. Rely on feature scaling to handle the drift.
D. Use a single validation set for model evaluation.

Solution

  1. Step 1: Identify the best approach for handling drift.

    Periodic model retraining helps adapt to both covariate and concept drift.

  2. Step 2: Eliminate wrong options.

    Static models (Use a static model trained on historical data.) and single validation sets (Use a single validation set for model evaluation.) don't adapt to drift. Feature scaling (Rely on feature scaling to handle the drift.) doesn't address drift.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Retrain models to adapt to drift [OK]
Hint: Retrain models to adapt to drift [OK]
Common Mistakes:
2. When choosing between polynomial features and interaction terms for a linear model, which is more appropriate when the dataset is small and why?
medium
A. Polynomial features, because they capture non-linear relationships better.
B. Interaction terms, because they increase model complexity less.
C. Polynomial features, because they require fewer parameters.
D. Interaction terms, because they are less prone to overfitting.

Solution

  1. Step 1: Consider dataset size and model complexity.

    Small datasets are prone to overfitting with complex models.

  2. Step 2: Evaluate the impact of each option.

    Polynomial features (A, C) increase complexity significantly, leading to overfitting. Interaction terms (B, D) add complexity but less than polynomial features, making them more suitable for small datasets.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Small data: prefer interaction terms to avoid overfitting. [OK]
Hint: Small data: prefer interaction terms to avoid overfitting. [OK]
Common Mistakes:
3. A search engine's NDCG score is unexpectedly low. What is a likely cause?
medium
A. The search engine is using an incorrect log base for DCG calculation.
B. The ideal DCG is calculated incorrectly.
C. The relevance scores are not normalized.
D. The search engine is returning too many irrelevant results at the top.

Solution

  1. Step 1: Identify NDCG sensitivity

    NDCG is sensitive to the position of relevant results; irrelevant results at the top lower the score.

  2. Step 2: Eliminate wrong options

    Normalization affects comparability, not score; incorrect ideal DCG or log base would affect all results, not just NDCG.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Top irrelevant results lower NDCG. [OK]
Hint: Top irrelevant results lower NDCG. [OK]
Common Mistakes:
4. In a fraud detection system, you need to balance precision and recall. Which metric would you choose and why?
hard
A. F1 Score, because it balances both precision and recall.
B. Precision, because false positives are more costly.
C. Recall, because missing a fraud case is more critical.
D. Accuracy, because it gives an overall performance measure.

Solution

  1. Step 1: Understand the need for balance.

    Fraud detection requires a balance between catching frauds and minimizing false alarms.

  2. Step 2: Choose the metric that balances precision and recall.

    F1 Score is the harmonic mean of precision and recall, providing a balance.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    F1 Score balances precision and recall. [OK]
Hint: F1 Score balances precision and recall. [OK]
Common Mistakes:
5. You are tasked with designing a system to handle label drift in a real-time recommendation engine. Which approach would be most effective?
hard
A. Implement a feedback loop to update the model with recent user interactions.
B. Use a static label mapping based on historical data.
C. Increase the size of the training dataset.
D. Apply feature selection to reduce dimensionality.

Solution

  1. Step 1: Identify the best approach for handling label drift.

    A feedback loop allows the model to adapt to changes in label distribution by learning from recent interactions.

  2. Step 2: Eliminate wrong options.

    Static mappings (Use a static label mapping based on historical data.) and increasing dataset size (Increase the size of the training dataset.) don't adapt to real-time changes. Feature selection (Apply feature selection to reduce dimensionality.) doesn't address label drift.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Feedback loop adapts to label drift [OK]
Hint: Feedback loop adapts to label drift [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard