Weight decay (L2 regularization) in PyTorch - Model Metrics & Evaluation

Metrics & Evaluation - Weight decay (L2 regularization)

Which metric matters for Weight Decay (L2 regularization) and WHY

Weight decay helps prevent overfitting by keeping model weights small. The key metrics to watch are validation loss and validation accuracy. If weight decay works well, validation loss should decrease or stay stable while training loss might be higher. This means the model generalizes better to new data.

Confusion matrix or equivalent visualization

Weight decay itself does not change the confusion matrix directly. But by reducing overfitting, it helps improve the confusion matrix on validation data. For example, a confusion matrix might look like this:

      Predicted
      |  TP=45  FP=5  |
      |  FN=10  TN=40 |
      Total samples = 100
    

Here, TP = true positives, FP = false positives, FN = false negatives, TN = true negatives. Weight decay helps improve these numbers by making the model less sensitive to noise.

Precision vs Recall tradeoff with Weight Decay

Weight decay reduces overfitting, which can improve both precision and recall on new data. But if weight decay is too strong, the model may underfit, lowering both precision and recall.

Example:

• Without weight decay: Precision=0.7, Recall=0.6 (overfitting, unstable)
• With moderate weight decay: Precision=0.75, Recall=0.7 (better generalization)
• With too much weight decay: Precision=0.6, Recall=0.5 (underfitting)

What "good" vs "bad" metric values look like for Weight Decay

Good: Validation loss close to training loss, stable or improving validation accuracy, balanced precision and recall.

Bad: Validation loss much higher than training loss (overfitting), or both losses high (underfitting). Precision or recall very low, showing poor generalization.

Common pitfalls with Weight Decay metrics

• Ignoring validation metrics and only looking at training loss can hide overfitting.
• Using too high weight decay can cause underfitting, making the model too simple.
• Data leakage can falsely improve validation metrics, hiding real overfitting.
• Confusing weight decay with dropout; they help differently.

Self-check: Your model has 98% accuracy but 12% recall on fraud. Is it good?

No, it is not good. High accuracy can be misleading if the data is imbalanced. A 12% recall means the model misses 88% of fraud cases, which is dangerous. Weight decay might help generalize better, but you need to improve recall for fraud detection.

Key Result

Weight decay improves generalization by balancing training and validation loss, helping avoid overfitting and underfitting.