Mutual information for feature selection in ML Python - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is mutual information in the context of feature selection?

Mutual information measures how much knowing one variable reduces uncertainty about another. In feature selection, it tells us how much information a feature gives about the target variable.

beginner

Why is mutual information useful for selecting features?

Because it captures any kind of relationship (not just linear) between a feature and the target, helping to pick features that truly help predict the target.

intermediate

How is mutual information different from correlation?

Correlation measures only linear relationships, while mutual information measures all types of relationships, including nonlinear ones.

beginner

What does a mutual information score of zero between a feature and the target mean?

It means the feature and the target are independent; knowing the feature does not give any information about the target.

intermediate

Name one limitation of using mutual information for feature selection.

Estimating mutual information from data can be tricky and may require many samples; also, it does not consider feature redundancy by itself.

What does mutual information measure between a feature and the target?

A The variance of the feature

B Only linear correlation

C The difference in means

D The amount of shared information

Mutual information measures how much information is shared between two variables, capturing all types of relationships.

If a feature has zero mutual information with the target, what does it mean?

A The feature perfectly predicts the target

B The feature has missing values

C The feature and target are independent

D The feature is highly correlated with the target

Zero mutual information means no dependency; the feature does not provide information about the target.

Which of these is a benefit of using mutual information for feature selection?

A It can detect nonlinear relationships

B It only detects linear relationships

C It ignores all relationships

D It requires no data

Mutual information captures both linear and nonlinear relationships between variables.

What is a common challenge when using mutual information in practice?

A Estimating it accurately needs enough data

B It always overfits the model

C It is easy to compute with few samples

D It only works for categorical features

Estimating mutual information well requires enough data to capture the true relationships.

Mutual information can help select features that are:

A Unrelated to the target

B Highly informative about the target

C Always redundant

D Only numeric

Features with high mutual information provide useful information about the target.

Explain in your own words what mutual information tells us about a feature and the target.

Think about how knowing the feature helps predict the target.

Describe one advantage and one limitation of using mutual information for feature selection.

Consider both what mutual information does well and what can be difficult.