Why regularization controls overfitting in PyTorch - Why Metrics Matter

Metrics & Evaluation - Why regularization controls overfitting

Which metric matters for this concept and WHY

When we talk about overfitting, the key metrics to watch are training loss and validation loss. Overfitting happens when training loss keeps going down but validation loss starts going up. Regularization helps by keeping the model simpler, so validation loss stays low too. This means the model learns patterns that work well on new data, not just the training data.

Confusion matrix or equivalent visualization (ASCII)

    Overfitting example confusion matrix:

          Predicted
          Pos   Neg
    True Pos  90    10
         Neg  30    70

    Here, the model fits training data well but makes more mistakes on new data.

    With regularization, errors on new data reduce:

          Predicted
          Pos   Neg
    True Pos  85    15
         Neg  15    85

    Regularization reduces false positives and false negatives by controlling complexity.
    

Precision vs Recall tradeoff with concrete examples

Regularization affects how complex the model is. A very complex model may have high precision but low recall because it memorizes training data and misses some true cases on new data. A simpler model with regularization balances precision and recall better by generalizing well.

For example, in spam detection:

• Without regularization: Model may mark many emails as spam (high recall) but also mark many good emails as spam (low precision).
• With regularization: Model better balances catching spam (recall) and not marking good emails as spam (precision).

What "good" vs "bad" metric values look like for this use case

Good: Training loss and validation loss both decrease and stay close. Precision and recall on validation data are balanced and high (e.g., >80%). This shows the model learned useful patterns without memorizing noise.

Bad: Training loss is very low but validation loss is high or increasing. Precision might be very high but recall very low, or vice versa. This means the model is overfitting and won't perform well on new data.

Metrics pitfalls (accuracy paradox, data leakage, overfitting indicators)

• Accuracy paradox: High accuracy can hide overfitting if data is imbalanced. Regularization helps by improving generalization, not just accuracy.
• Data leakage: If validation data leaks into training, metrics look good but model overfits. Regularization cannot fix this.
• Overfitting indicators: Large gap between training and validation loss, or very high training accuracy but low validation accuracy.

Self-check: Your model has 98% accuracy but 12% recall on fraud. Is it good?

No, this model is not good for fraud detection. The 98% accuracy is misleading because fraud cases are rare. The 12% recall means the model misses 88% of fraud cases, which is dangerous. Regularization alone won't fix this; you need to improve recall by adjusting the model or data.

Key Result

Regularization helps keep training and validation losses close, reducing overfitting and improving model generalization.