Recursive feature elimination in ML Python

Introduction

Recursive feature elimination helps find the most important features in your data by removing less useful ones step by step.

When you have many features and want to keep only the most useful ones.

When you want to improve model speed by using fewer features.

When you want to understand which features affect your model's decisions the most.

When you want to reduce noise from irrelevant features.

When you want to avoid overfitting by simplifying your model.

Syntax

ML Python

from sklearn.feature_selection import RFE
from sklearn.linear_model import LogisticRegression

model = LogisticRegression()
rfe = RFE(estimator=model, n_features_to_select=number_of_features)
rfe.fit(X, y)
selected_features = rfe.support_

estimator is the model used to judge feature importance.

n_features_to_select is how many features you want to keep.

Examples

Using a decision tree to select top 3 features.

ML Python

from sklearn.feature_selection import RFE
from sklearn.tree import DecisionTreeClassifier

model = DecisionTreeClassifier()
rfe = RFE(estimator=model, n_features_to_select=3)
rfe.fit(X, y)
print(rfe.support_)

Selecting only the single most important feature.

ML Python

from sklearn.feature_selection import RFE
from sklearn.linear_model import LogisticRegression

model = LogisticRegression()
rfe = RFE(estimator=model, n_features_to_select=1)
rfe.fit(X, y)
print(rfe.support_)

Edge case: selecting zero features will raise an error.

ML Python

from sklearn.feature_selection import RFE
from sklearn.linear_model import LogisticRegression

model = LogisticRegression()
rfe = RFE(estimator=model, n_features_to_select=0)
rfe.fit(X, y)
print(rfe.support_)

Sample Model

This program loads the iris flower data, uses recursive feature elimination to keep the two most important features, and then predicts the flower type using only those features.

ML Python

from sklearn.datasets import load_iris
from sklearn.feature_selection import RFE
from sklearn.linear_model import LogisticRegression

# Load data
iris = load_iris()
X = iris.data
y = iris.target

# Create model
model = LogisticRegression(max_iter=200)

# Use RFE to select top 2 features
rfe = RFE(estimator=model, n_features_to_select=2)
rfe.fit(X, y)

# Show which features were selected
print('Selected features mask:', rfe.support_)

# Show ranking of features (1 means selected)
print('Feature ranking:', rfe.ranking_)

# Predict using selected features
X_selected = rfe.transform(X)
predictions = rfe.estimator_.predict(X_selected)

# Show first 5 predictions
print('First 5 predictions:', predictions[:5])

Important Notes

Time complexity depends on the estimator and number of features; it can be slow for many features.

Space complexity is similar to the estimator's requirements.

Common mistake: setting n_features_to_select to zero or more than total features causes errors.

Use RFE when you want to reduce features based on model importance; use other methods if you want filter-based selection.

Summary

Recursive feature elimination removes less important features step by step.

It helps improve model speed and understanding by keeping only key features.

You need to choose a model to judge feature importance and how many features to keep.