Bird
Raised Fist0

Feature Selection - Filter, Wrapper, Embedded Methods

THEORYintermediateInterview freq: medium
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
⚖️
Filter Methods vs Wrapper Methods

In a dataset with 100K features and a limited compute budget, would you choose filter or wrapper methods for feature selection? Justify your choice.

📊
Comparison
DimensionFilter MethodsWrapper Methods
ScalabilityHighLow
AccuracyMediumHigh
Computational CostLowHigh
Feature InteractionsLowHigh
Ease of ImplementationHighMedium
✅
Verdict
Default to filter methods for datasets with over 10K features and limited compute resources. Switch to wrapper methods when feature interactions are critical and you have sufficient computational budget.
💡
Key Insight
Filter methods excel in scalability, while wrapper methods capture complex interactions at a computational cost.
⚠️
Interview Trap
❌ Wrong: Always use wrapper methods for better accuracy.
Ignoring the computational cost and scalability issues with large feature sets.
✅ Consider the trade-off between computational cost and the need for capturing feature interactions. Use wrapper methods when accuracy is paramount and resources allow.
Filter Methods
"Scalability to large datasets due to low computational cost"
Weaknesses
  • Ignores feature interactions, leading to suboptimal feature sets
  • May select irrelevant features if they correlate with the target by chance
  • Less effective for non-linear relationships
Ideal When
  • n > 100K
  • feature count > 10K
  • limited compute resources
  • need for quick preprocessing
Netflix uses filter methods for initial feature reduction in their recommendation systems to handle massive datasets efficiently.
Wrapper Methods
"Ability to capture feature interactions and model-specific relevance"
Weaknesses
  • Computationally expensive, especially with large feature sets
  • Prone to overfitting if not properly regularized
  • Requires a model to evaluate feature subsets, increasing complexity
Ideal When
  • feature count < 1K
  • ample compute resources
  • need for high accuracy
  • non-linear relationships present
Google uses wrapper methods in their search algorithms to fine-tune feature sets for specific ranking models, where accuracy is critical.
🐍
Filter Methods - Example
▼ Show
Python
import numpy as np
from sklearn.feature_selection import SelectKBest, f_classif
X = np.random.rand(1000, 10000)
y = np.random.randint(0, 2, 1000)
selector = SelectKBest(score_func=f_classif, k=50)
X_new = selector.fit_transform(X, y)
🐍
Wrapper Methods - Example
▼ Show
Python
from sklearn.feature_selection import RFE
from sklearn.linear_model import LogisticRegression
X = np.random.rand(100, 20)
y = np.random.randint(0, 2, 100)
model = LogisticRegression()
rfe = RFE(model, n_features_to_select=5)
X_new = rfe.fit_transform(X, y)
✅
Use Filter Methods When
  • •n > 100K
  • •feature count > 10K
  • •compute budget < 10 hours
  • •need for quick preprocessing
✅
Use Wrapper Methods When
  • •feature count < 1K
  • •compute budget > 24 hours
  • •high accuracy is critical
  • •non-linear relationships are expected
↳
Nuances
  • Filter methods may miss important features that only show relevance in combination with others.
  • Wrapper methods can overfit if the model used for evaluation is too complex or not regularized.
  • In high-dimensional spaces, filter methods can provide a good starting point before applying more computationally intensive methods.

Practice

(1/5)
1. In the context of cross-validation, which method is most appropriate for imbalanced datasets?
easy
A. Time Series Split
B. K-Fold Cross-Validation
C. Stratified K-Fold Cross-Validation
D. Leave-One-Out Cross-Validation

Solution

  1. Step 1: Identify the need for balanced class distribution.

    Imbalanced datasets require maintaining class distribution in each fold.

  2. Step 2: Eliminate wrong options.

    K-Fold does not ensure class balance (A), Time Series Split is for temporal data (C), Leave-One-Out is computationally expensive and doesn't address imbalance (D).

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Stratified keeps class balance. [OK]
Hint: Stratified keeps class balance. [OK]
Common Mistakes:
2. What will be the output shape of the following code snippet using K-Fold cross-validation with 5 splits on a dataset of 100 samples?
from sklearn.model_selection import KFold
kf = KFold(n_splits=5)
for train_index, test_index in kf.split(range(100)):
    print(len(test_index))
easy
A. 20, 20, 20, 20, 20
B. 25, 25, 25, 25
C. 20, 20, 20, 20, 20, 20
D. 20, 20, 20, 20

Solution

  1. Step 1: Calculate the number of samples per fold.

    100 samples divided by 5 splits results in 20 samples per fold.

  2. Step 2: Eliminate wrong options.

    25 samples per fold (B) is incorrect, 6 outputs (C) is incorrect, 4 outputs (D) is incorrect.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    100 samples, 5 splits = 20 per fold. [OK]
Hint: 100 samples, 5 splits = 20 per fold. [OK]
Common Mistakes:
3. What will be the output of the following Python code snippet using XGBoost?
import xgboost as xgb
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split

data = load_iris()
X_train, X_test, y_train, y_test = train_test_split(data.data, data.target, test_size=0.2, random_state=42)
dtrain = xgb.DMatrix(X_train, label=y_train)
dtest = xgb.DMatrix(X_test, label=y_test)

params = {'objective': 'multi:softprob', 'num_class': 3}
model = xgb.train(params, dtrain, num_boost_round=10)
preds = model.predict(dtest)
print(preds.shape)
easy
A. (120, 3)
B. (30, 3)
C. (30,)
D. (120,)

Solution

  1. Step 1: Understand the code

    The code splits the Iris dataset into training and testing sets, with 20% for testing, resulting in 30 test samples.

  2. Step 2: Analyze the prediction output

    The model predicts probabilities for 3 classes, so the output shape is (30, 3).

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Test set size = 30, 3 classes [OK]
Hint: Test set size = 30, 3 classes [OK]
Common Mistakes:
4. In the context of K-Means clustering, which initialization method is most likely to avoid poor local minima?
easy
A. K-Means++ initialization
B. Random initialization
C. Using centroids from a previous run
D. Initializing all centroids to the same point

Solution

  1. Step 1: Understand initialization impact

    K-Means++ initialization helps in spreading initial centroids to avoid poor local minima.

  2. Step 2: Eliminate wrong options

    Random initialization can lead to poor local minima. Using centroids from a previous run might not improve if the previous run was suboptimal. Initializing all centroids to the same point is clearly incorrect.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    K-Means++ spreads centroids well [OK]
Hint: K-Means++ spreads centroids well [OK]
Common Mistakes:
5. For a dataset with significant outliers, which scaling method is more appropriate and why?
medium
A. Standardization, because it scales data to a fixed range.
B. Normalization, because it scales data to a fixed range.
C. Normalization, because it handles outliers better.
D. Standardization, because it is less affected by outliers.

Solution

  1. Step 1: Consider the effect of outliers on scaling methods.

    Normalization is sensitive to outliers as it scales data to a fixed range.

  2. Step 2: Choose the method less affected by outliers.

    Standardization is less affected because it centers data around the mean and scales by standard deviation.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Standardize for outliers, less sensitive. [OK]
Hint: Standardize for outliers, less sensitive. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard