Mutual information for feature selection in ML Python - Model Metrics & Evaluation

Metrics & Evaluation - Mutual information for feature selection

Which metric matters for this concept and WHY

Mutual information measures how much knowing one thing helps you know another. For feature selection, it tells us how much a feature and the target share information. The higher the mutual information, the more useful the feature is for predicting the target. This helps pick features that really matter and ignore noise.

Confusion matrix or equivalent visualization (ASCII)

Mutual information is not based on a confusion matrix but on probabilities. Imagine a table showing how often each feature value pairs with each target value:

    | Feature Value | Target=0 Count | Target=1 Count |
    |---------------|----------------|----------------|
    |       A       |       30       |       10       |
    |       B       |       20       |       40       |
    

Mutual information uses these counts to calculate how much knowing the feature reduces uncertainty about the target.

Precision vs Recall (or equivalent tradeoff) with concrete examples

Mutual information helps decide which features to keep. A tradeoff is between keeping many features (high recall of useful info) and keeping only the best (high precision of relevant features).

For example, if you keep too many features with low mutual information, your model may be slow and confused by noise (low precision). If you keep too few, you might miss important signals (low recall).

Balancing this tradeoff means selecting features with mutual information above a threshold that keeps most useful info but removes noise.

What "good" vs "bad" metric values look like for this use case

Good mutual information values are higher numbers showing strong connection between feature and target. For example, a mutual information of 0.5 or above (on a scale from 0 to 1) means the feature shares a lot of info with the target.

Bad values are close to 0, meaning the feature gives almost no useful info about the target. Such features can be dropped safely.

Remember, mutual information is always >= 0. Zero means no relationship.

Metrics pitfalls (accuracy paradox, data leakage, overfitting indicators)

• Ignoring feature redundancy: Two features can both have high mutual information but carry the same info. Selecting both adds no benefit.
• Data leakage: If the feature leaks future info about the target, mutual information will be high but model will fail in real use.
• Overfitting: Selecting features based on mutual information from the test set can cause overfitting. Always compute on training data only.
• Ignoring feature interactions: Mutual information looks at one feature at a time. Some features may be weak alone but strong together.

Your model has 98% accuracy but 12% recall on fraud. Is it good?

No, it is not good for fraud detection. Even though accuracy is high, the model misses 88% of fraud cases (low recall). For fraud, catching as many frauds as possible is critical, so recall matters more than accuracy.

This shows why choosing the right metric matters. High accuracy can be misleading if the data is imbalanced or the goal is to catch rare events.

Key Result

Mutual information quantifies how much a feature tells us about the target, guiding effective feature selection by highlighting informative features.