Feature selection methods in ML Python - Model Metrics & Evaluation

Metrics & Evaluation - Feature selection methods

Which metric matters for Feature Selection and WHY

Feature selection helps pick the most useful data parts for a model. The key metrics to check are model accuracy, precision, and recall after selecting features. These show if the chosen features help the model make better predictions.

Also, watch model training time and complexity. Good feature selection reduces these, making the model faster and simpler.

Confusion Matrix Example After Feature Selection

      Actual \ Predicted | Positive | Negative
      -------------------|----------|---------
      Positive           |    80    |   20
      Negative           |    10    |   90
    

From this matrix:

• True Positives (TP) = 80
• False Positives (FP) = 10
• True Negatives (TN) = 90
• False Negatives (FN) = 20

Precision = 80 / (80 + 10) = 0.89

Recall = 80 / (80 + 20) = 0.80

These numbers show how well the model performs with the selected features.

Precision vs Recall Tradeoff in Feature Selection

Choosing features affects precision and recall differently:

• High precision means fewer false alarms. Useful when false positives are costly, like spam filters.
• High recall means catching most real cases. Important in health checks, like cancer detection.

Feature selection can improve one but hurt the other. For example, removing features might reduce false positives (better precision) but miss some true cases (lower recall).

Good vs Bad Metric Values for Feature Selection

Good:

• Accuracy above baseline (better than random guessing)
• Precision and recall balanced and high (e.g., both above 0.8)
• Reduced training time and simpler model

Bad:

• Accuracy close to random (e.g., 50% for binary)
• Very low precision or recall (below 0.5)
• Model complexity remains high despite feature selection

Common Pitfalls in Feature Selection Metrics

• Accuracy paradox: High accuracy can hide poor recall or precision if classes are imbalanced.
• Data leakage: Using future or test data features can falsely boost metrics.
• Overfitting: Selecting features that fit training data noise leads to poor real-world results.
• Ignoring metric tradeoffs: Focusing only on accuracy without checking precision and recall can mislead.

Self-Check Question

Your model after feature selection has 98% accuracy but only 12% recall on the positive class (e.g., fraud). Is it good for production? Why or why not?

Answer: No, it is not good. The very low recall means the model misses most positive cases (fraud). Even with high accuracy, the model fails to catch important cases, which is critical in fraud detection.

Key Result

Feature selection improves model performance by balancing accuracy, precision, recall, and reducing complexity.