Mutual information for feature selection in ML Python - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding Mutual Information in Feature Selection

Which statement best describes the role of mutual information in feature selection?

A It quantifies the amount of shared information between a feature and the target variable, capturing any kind of dependency.

B It measures the linear correlation between features and the target variable.

C It ranks features based on their variance across the dataset.

D It removes features that have missing values in the dataset.

💡 Hint

Think about how mutual information captures relationships beyond just linear ones.

Mutual information measures how much knowing the value of one variable reduces uncertainty about another, capturing all types of dependencies, not just linear correlations.

❓ Predict Output

intermediate

Output of Mutual Information Calculation

What is the output of the following Python code that calculates mutual information between features and a binary target?

ML Python

from sklearn.feature_selection import mutual_info_classif
import numpy as np

X = np.array([[1, 2, 3], [1, 3, 3], [0, 2, 1], [0, 3, 1]])
y = np.array([0, 1, 0, 1])

mi = mutual_info_classif(X, y, discrete_features=[True, True, True], random_state=0)
print([round(v, 2) for v in mi])

A [0.0, 0.0, 0.0]

B [0.19, 0.0, 0.0]

C [0.0, 0.19, 0.0]

D [0.19, 0.19, 0.0]

💡 Hint

Mutual information is non-negative and measures dependency; check which feature varies with the target.

The second feature varies with the target, so it has positive mutual information; the others do not.

❓ Model Choice

advanced

Choosing Features Based on Mutual Information

You have 10 features and their mutual information scores with the target. Which approach best selects features to improve model performance?

A Select the top 3 features with the highest mutual information scores only.

B Select features with mutual information scores above zero, regardless of redundancy between features.

C Select features randomly to avoid bias from mutual information scores.

D Select features with high mutual information scores and low mutual information among themselves to reduce redundancy.

💡 Hint

Consider both relevance to target and redundancy among features.

Choosing features that are both relevant and not redundant helps improve model performance by providing diverse information.

❓ Hyperparameter

advanced

Hyperparameter Impact on Mutual Information Estimation

When using mutual_info_classif from scikit-learn, which hyperparameter affects the smoothness of the mutual information estimate for continuous features?

A <code>n_neighbors</code>

B <code>random_state</code>

C <code>discrete_features</code>

D <code>copy</code>

💡 Hint

Think about parameters controlling neighborhood size in nearest neighbor estimation.

The n_neighbors parameter controls the number of neighbors used in the k-nearest neighbors method for estimating mutual information, affecting smoothness.

🔧 Debug

expert

Debugging Mutual Information Calculation Error

What error will the following code raise when calculating mutual information, and why?

from sklearn.feature_selection import mutual_info_classif
import numpy as np

X = np.array([[1.5, 2.3], [3.1, 4.7], [5.2, 6.8]])
y = np.array([0, 1, 0])

mi = mutual_info_classif(X, y, discrete_features=True)
print(mi)

A RuntimeWarning due to division by zero in mutual information calculation.

B TypeError because <code>discrete_features</code> must be a boolean or array-like, not a single boolean when X has continuous features.

C No error; the code runs and outputs mutual information scores.

D ValueError because the number of samples in X and y do not match.

💡 Hint

Check the type and shape of discrete_features parameter relative to input data.

Setting discrete_features=True treats all features as discrete, but with continuous data this causes a TypeError; it should be a boolean array or False for continuous features.