Mutual information for feature selection in ML Python - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is mutual information in the context of feature selection?

Mutual information measures how much knowing one variable reduces uncertainty about another. In feature selection, it tells us how much information a feature gives about the target variable.

beginner

Why is mutual information useful for selecting features?

Because it captures any kind of relationship (not just linear) between a feature and the target, helping to pick features that truly help predict the target.

intermediate

How is mutual information different from correlation?

Correlation measures only linear relationships, while mutual information measures all types of relationships, including nonlinear ones.

beginner

What does a mutual information score of zero between a feature and the target mean?

It means the feature and the target are independent; knowing the feature does not give any information about the target.

intermediate

Name one limitation of using mutual information for feature selection.

Estimating mutual information from data can be tricky and may require many samples; also, it does not consider feature redundancy by itself.

What does mutual information measure between a feature and the target?

A The variance of the feature

B Only linear correlation

C The difference in means

D The amount of shared information

Mutual information measures how much information is shared between two variables, capturing all types of relationships.

If a feature has zero mutual information with the target, what does it mean?

A The feature perfectly predicts the target

B The feature has missing values

C The feature and target are independent

D The feature is highly correlated with the target

Zero mutual information means no dependency; the feature does not provide information about the target.

Which of these is a benefit of using mutual information for feature selection?

A It can detect nonlinear relationships

B It only detects linear relationships

C It ignores all relationships

D It requires no data

Mutual information captures both linear and nonlinear relationships between variables.

What is a common challenge when using mutual information in practice?

A Estimating it accurately needs enough data

B It always overfits the model

C It is easy to compute with few samples

D It only works for categorical features

Estimating mutual information well requires enough data to capture the true relationships.

Mutual information can help select features that are:

A Unrelated to the target

B Highly informative about the target

C Always redundant

D Only numeric

Features with high mutual information provide useful information about the target.

Explain in your own words what mutual information tells us about a feature and the target.

Think about how knowing the feature helps predict the target.

Describe one advantage and one limitation of using mutual information for feature selection.

Consider both what mutual information does well and what can be difficult.

Practice

1. What does mutual information measure in feature selection?

easy

A. The amount of shared information between a feature and the target variable

B. The correlation coefficient between two features

C. The difference between feature means

D. The number of missing values in a feature

Solution

1. Step 1: Understand mutual information concept

   Mutual information measures how much knowing one variable reduces uncertainty about another.

2. Step 2: Apply to feature selection context

   In feature selection, it measures how much information a feature shares with the target variable.

3. Final Answer:

   The amount of shared information between a feature and the target variable -> Option A

4. Quick Check:

   Mutual information = shared info [OK]

Hint: Mutual info = shared info between feature and target [OK]

Common Mistakes:

• Confusing mutual information with correlation
• Thinking it measures missing data
• Assuming it measures difference in means

2. Which Python function is used to compute mutual information for classification tasks?

easy

A. mutual_info_classif

B. mutual_info_regression

C. mutual_info_score

D. mutual_info_classifier

Solution

1. Step 1: Recall mutual information functions in sklearn

   For classification, sklearn provides mutual_info_classif.

2. Step 2: Differentiate from regression function

   mutual_info_regression is for regression, not classification.

3. Final Answer:

   mutual_info_classif -> Option A

4. Quick Check:

   Classification uses mutual_info_classif [OK]

Hint: Classification uses mutual_info_classif function [OK]

Common Mistakes:

• Using mutual_info_regression for classification
• Confusing function names
• Assuming mutual_info_score exists in sklearn

3. Given this code snippet, what is the output?

from sklearn.feature_selection import mutual_info_classif
import numpy as np
X = np.array([[1, 2], [2, 3], [3, 4], [4, 5]])
y = np.array([0, 1, 0, 1])
mi = mutual_info_classif(X, y, discrete_features=[True, True])
print(np.round(mi, 2))

medium

A. [0.0 0.0]

B. [0.69 0.0]

C. [0.0 0.69]

D. [0.69 0.69]

Solution

1. Step 1: Understand input data and parameters

   X has two discrete features, y is binary. Using mutual_info_classif with discrete_features=True for both.

2. Step 2: Calculate mutual information values

   Both features vary similarly with y, so both have similar mutual information around 0.69 (close to ln(2)).

3. Final Answer:

   [0.69 0.69] -> Option D

4. Quick Check:

   Both features share info with y ~0.69 [OK]

Hint: Discrete features with binary target give ~0.69 MI if informative [OK]

Common Mistakes:

• Assuming zero mutual information for all features
• Mixing up discrete_features parameter
• Rounding errors in output

4. Identify the error in this code snippet for mutual information feature selection:

from sklearn.feature_selection import mutual_info_classif
X = [[1, 2], [2, 3], [3, 4]]
y = [0, 1, 0]
mi = mutual_info_classif(X, y)
print(mi)

medium

A. y should be a 2D array, not 1D

B. X should be a numpy array, not a list of lists

C. mutual_info_classif requires discrete_features parameter

D. mutual_info_classif cannot handle integer data

Solution

1. Step 1: Check input data types

   mutual_info_classif expects numpy arrays or similar, not plain Python lists.

2. Step 2: Identify error cause

   Passing list of lists for X can cause unexpected behavior or errors; converting to numpy array fixes this.

3. Final Answer:

   X should be a numpy array, not a list of lists -> Option B

4. Quick Check:

   Use numpy arrays for X [OK]

Hint: Always convert input data to numpy arrays before sklearn functions [OK]

Common Mistakes:

• Thinking y must be 2D
• Assuming discrete_features is always required
• Believing mutual_info_classif rejects integer data

5. You have a dataset with 10 features. After computing mutual information scores, you find two features have the highest scores but are highly correlated with each other. What is the best approach to select features?

hard

A. Select both features because they have the highest mutual information

B. Select features randomly to avoid bias

C. Select only one of the two correlated features with the highest mutual information

D. Discard both features to avoid redundancy

Solution

1. Step 1: Understand mutual information and correlation

   High mutual information means features are informative, but high correlation means redundancy.

2. Step 2: Choose features to reduce redundancy

   To avoid redundant information, select only one of the correlated features with the highest mutual information.

3. Final Answer:

   Select only one of the two correlated features with the highest mutual information -> Option C

4. Quick Check:

   Pick one correlated feature with highest MI [OK]

Hint: Avoid redundant features by picking one with highest MI [OK]

Common Mistakes:

• Selecting both correlated features causing redundancy
• Discarding informative features unnecessarily
• Choosing features randomly without criteria