Creating interaction features in ML Python - Evaluation Workflow

Metrics & Evaluation - Creating interaction features

Which metric matters for creating interaction features and WHY

When we create interaction features, we want to see if they help the model learn better. The main metrics to check are validation accuracy or validation loss. These show if the model predicts better on new data, not just the training data.

If the interaction features improve these metrics, it means they add useful information. If not, they might just add noise or make the model too complex.

Confusion matrix example

Suppose we have a classification task. Here is a confusion matrix before and after adding interaction features:

Before interaction features:
| TP=40 | FP=10 |
| FN=15 | TN=35 |

After interaction features:
| TP=45 | FP=8  |
| FN=10 | TN=37 |
    

Adding interaction features increased true positives and true negatives, and reduced false negatives and false positives. This means better predictions.

Precision vs Recall tradeoff with interaction features

Interaction features can help balance precision and recall. For example:

• Precision measures how many predicted positives are correct.
• Recall measures how many actual positives are found.

If interaction features help the model find more true positives without adding many false positives, recall and precision both improve.

But if interaction features cause the model to predict too many positives, precision may drop even if recall rises.

We want to find interaction features that improve both or at least keep a good balance.

Good vs Bad metric values for interaction features

Good: After adding interaction features, validation accuracy increases, validation loss decreases, and precision and recall improve or stay stable.

Bad: Validation accuracy drops or stays the same, loss increases, or precision and recall get worse. This means interaction features are not helping.

Also watch for overfitting: if training accuracy improves but validation accuracy drops, interaction features might be too complex.

Common pitfalls when evaluating interaction features

• Overfitting: Interaction features can make the model too complex, fitting noise instead of real patterns.
• Data leakage: Creating interaction features using future or test data can give false good results.
• Ignoring validation: Only checking training metrics can mislead you about feature usefulness.
• Too many features: Adding many interaction features can slow training and confuse the model.

Self-check question

Your model has 98% accuracy but only 12% recall on fraud cases after adding interaction features. Is it good for production?

Answer: No. Even though accuracy is high, recall is very low. This means the model misses most fraud cases, which is bad for fraud detection. Interaction features did not help find fraud well.

Key Result

Interaction features should improve validation accuracy and balance precision and recall without causing overfitting.