Mutual information for feature selection in ML Python

Introduction

Mutual information helps us find which features give the most useful information about the target. It helps pick the best features to improve model accuracy.

When you want to select important features before training a model.

When you have many features and want to reduce them to save time.

When you want to understand which features relate most to the target.

When you want to improve model performance by removing irrelevant features.

Syntax

ML Python

from sklearn.feature_selection import mutual_info_classif

mi = mutual_info_classif(X, y)

# X: feature data (2D array), y: target labels (1D array)
# mi: array of mutual information scores for each feature

mutual_info_classif is for classification tasks.

For regression, use mutual_info_regression instead.

Examples

Calculate mutual information scores for all features and print them.

ML Python

from sklearn.feature_selection import mutual_info_classif

mi_scores = mutual_info_classif(X, y)
print(mi_scores)

Automatically detect which features are discrete or continuous.

ML Python

from sklearn.feature_selection import mutual_info_classif

mi_scores = mutual_info_classif(X, y, discrete_features='auto')

Use 5 neighbors to estimate mutual information, which can affect smoothness.

ML Python

from sklearn.feature_selection import mutual_info_classif

mi_scores = mutual_info_classif(X, y, n_neighbors=5)

Sample Model

This program loads the iris flower dataset, calculates mutual information scores for each feature, and prints the scores. Higher scores mean the feature is more informative about the flower type.

ML Python

from sklearn.datasets import load_iris
from sklearn.feature_selection import mutual_info_classif

# Load iris dataset
data = load_iris()
X = data.data
y = data.target

# Calculate mutual information scores
mi_scores = mutual_info_classif(X, y)

# Print feature names with their scores
for name, score in zip(data.feature_names, mi_scores):
    print(f"{name}: {score:.4f}")

Important Notes

Mutual information measures how much knowing a feature reduces uncertainty about the target.

It works well for both categorical and continuous features.

Scores are always non-negative; higher means more useful.

Summary

Mutual information helps pick features that share the most information with the target.

Use mutual_info_classif for classification and mutual_info_regression for regression.

Higher mutual information scores mean more important features.