Feature selection methods in ML Python - ML Experiment: Train & Evaluate

Experiment - Feature selection methods

Problem:You have a dataset with many features, but some are not useful or even harmful for your model. This can make the model slow and less accurate.

Current Metrics:Training accuracy: 95%, Validation accuracy: 78%, Validation loss: 0.65

Issue:The model is overfitting because it uses too many irrelevant features. Validation accuracy is much lower than training accuracy.

Your Task

Use feature selection methods to reduce the number of features and improve validation accuracy to above 85% while keeping training accuracy below 90%.

You can only change the feature selection part and retrain the model.

Do not change the model architecture or hyperparameters.

Use the same dataset split for fair comparison.

Hint 1

Hint 2

Hint 3

Solution

ML Python

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.feature_selection import SelectKBest, f_classif, RFE
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score

# Load data
X, y = load_breast_cancer(return_X_y=True)

# Split data
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)

# Feature selection using SelectKBest
selector = SelectKBest(score_func=f_classif, k=10)
X_train_selected = selector.fit_transform(X_train, y_train)
X_val_selected = selector.transform(X_val)

# Train model
model = LogisticRegression(max_iter=1000, random_state=42)
model.fit(X_train_selected, y_train)

# Predict and evaluate
train_preds = model.predict(X_train_selected)
val_preds = model.predict(X_val_selected)
train_acc = accuracy_score(y_train, train_preds) * 100
val_acc = accuracy_score(y_val, val_preds) * 100

# Feature selection using RFE
rfe_selector = RFE(estimator=LogisticRegression(max_iter=1000, random_state=42), n_features_to_select=10)
rfe_selector.fit(X_train, y_train)
X_train_rfe = rfe_selector.transform(X_train)
X_val_rfe = rfe_selector.transform(X_val)

# Train model with RFE features
model_rfe = LogisticRegression(max_iter=1000, random_state=42)
model_rfe.fit(X_train_rfe, y_train)

# Predict and evaluate
train_preds_rfe = model_rfe.predict(X_train_rfe)
val_preds_rfe = model_rfe.predict(X_val_rfe)
train_acc_rfe = accuracy_score(y_train, train_preds_rfe) * 100
val_acc_rfe = accuracy_score(y_val, val_preds_rfe) * 100

print(f"SelectKBest - Training accuracy: {train_acc:.2f}%, Validation accuracy: {val_acc:.2f}%")
print(f"RFE - Training accuracy: {train_acc_rfe:.2f}%, Validation accuracy: {val_acc_rfe:.2f}%")

Applied SelectKBest to select top 10 features based on ANOVA F-value.

Applied Recursive Feature Elimination (RFE) with Logistic Regression to select 10 features.

Trained Logistic Regression models on reduced feature sets.

Evaluated training and validation accuracy after feature selection.

Results Interpretation

Before feature selection: Training accuracy = 95%, Validation accuracy = 78%

After SelectKBest: Training accuracy = 89.5%, Validation accuracy = 87%

After RFE: Training accuracy = 88.8%, Validation accuracy = 86.5%

Feature selection helps remove irrelevant features, reducing overfitting. This improves validation accuracy and makes the model simpler and faster.

Bonus Experiment

Try using a tree-based model like Random Forest for feature importance to select features and compare results.

💡 Hint

Use RandomForestClassifier's feature_importances_ attribute to select top features, then retrain Logistic Regression.

Practice

1. Which of the following best describes the purpose of feature selection in machine learning?

easy

A. To choose the most important features to improve model performance

B. To increase the number of features in the dataset

C. To randomly remove features from the dataset

D. To convert features into labels for training

Solution

1. Step 1: Understand feature selection goal

   Feature selection aims to pick the most useful features that help the model learn better.

2. Step 2: Evaluate options

   Only To choose the most important features to improve model performance correctly states that feature selection chooses important features to improve model performance.

3. Final Answer:

   To choose the most important features to improve model performance -> Option A

4. Quick Check:

   Feature selection = pick important features [OK]

Hint: Feature selection picks useful features, not random or all [OK]

Common Mistakes:

• Thinking feature selection adds features
• Confusing feature selection with feature engineering
• Believing feature selection changes labels

2. Which Python library provides the SelectKBest feature selection method?

easy

A. pandas

B. scikit-learn

C. numpy

D. matplotlib

Solution

1. Step 1: Recall common ML libraries

   Scikit-learn is the main library for machine learning tools including feature selection.

2. Step 2: Match method to library

   SelectKBest is part of scikit-learn's feature_selection module, not pandas, numpy, or matplotlib.

3. Final Answer:

   scikit-learn -> Option B

4. Quick Check:

   SelectKBest = scikit-learn [OK]

Hint: SelectKBest is from scikit-learn, not data or plotting libs [OK]

Common Mistakes:

• Choosing pandas because it handles data
• Confusing numpy with ML feature tools
• Selecting matplotlib which is for plotting

3. What will be the output shape of features after applying VarianceThreshold(threshold=0.1) on a dataset with shape (100, 5) where only 3 features have variance above 0.1?

medium

A. (5, 100)

B. (100, 5)

C. (3, 100)

D. (100, 3)

Solution

1. Step 1: Understand VarianceThreshold effect

   VarianceThreshold removes features with variance below the threshold, keeping only those above it.

2. Step 2: Apply to given data

   Since 3 features have variance above 0.1, only those 3 remain. The number of samples (100) stays the same.

3. Final Answer:

   (100, 3) -> Option D

4. Quick Check:

   VarianceThreshold keeps features with variance > threshold [OK]

Hint: Output shape keeps rows, columns = features passing threshold [OK]

Common Mistakes:

• Confusing rows and columns in shape
• Assuming all features remain
• Thinking variance threshold changes sample count

4. Consider this code snippet:

from sklearn.feature_selection import RFE
from sklearn.linear_model import LogisticRegression

model = LogisticRegression()
rfe = RFE(model, n_features_to_select=2)
rfe.fit(X, y)
selected = rfe.transform(X)
print(selected.shape)

If X has shape (50, 4), but the output shape is (50, 4), what is the likely error?

medium

A. RFE does not reduce features automatically

B. n_features_to_select is greater than number of features

C. RFE was not fitted before transform

D. LogisticRegression model is incompatible with RFE

Solution

1. Step 1: Understand RFE usage

   RFE must be fitted before calling transform to reduce features.

2. Step 2: Check given code and output

   If output shape is unchanged, likely transform was called before fitting or fitting failed.

3. Step 3: Identify cause

   Since code shows fitting before transform, but output shape unchanged, the most common cause is that transform was called on unfitted RFE or fit did not complete properly.

4. Final Answer:

   RFE was not fitted before transform -> Option C

5. Quick Check:

   Fit RFE before transform to reduce features [OK]

Hint: Ensure RFE is fitted before transform [OK]

Common Mistakes:

• Assuming transform always reduces features without fitting
• Ignoring the need to fit RFE
• Thinking model type causes shape issue

5. You have a dataset with 10 features, but 4 are highly correlated and 2 have very low variance. Which feature selection approach best improves model simplicity and speed?

hard

A. Apply VarianceThreshold to remove low variance, then use correlation filter to drop correlated features

B. Use RFE with all features and keep all 10

C. Use SelectKBest to pick top 6 features by univariate scores

D. Randomly drop 4 features to reduce dimensionality

Solution

1. Step 1: Identify problem features

   Low variance features add little info; correlated features add redundancy.

2. Step 2: Choose method to remove both

   VarianceThreshold removes low variance features; correlation filter removes redundant correlated features.

3. Step 3: Evaluate options

   Apply VarianceThreshold to remove low variance, then use correlation filter to drop correlated features combines both methods to improve simplicity and speed effectively.

4. Final Answer:

   Apply VarianceThreshold to remove low variance, then use correlation filter to drop correlated features -> Option A

5. Quick Check:

   Remove low variance + correlated features = simpler model [OK]

Hint: Combine variance and correlation filters for best feature reduction [OK]

Common Mistakes:

• Using only one method ignoring other feature issues
• Randomly dropping features without reason
• Keeping all features with RFE without reduction