Validation loop in PyTorch - Model Metrics & Evaluation

Metrics & Evaluation - Validation loop

Which metric matters for Validation Loop and WHY

The validation loop helps us check how well our model works on new data it hasn't seen before. The key metrics here are validation loss and validation accuracy. Validation loss tells us how far off the model's predictions are from the true answers on the validation set. Validation accuracy shows the percentage of correct predictions. These metrics matter because they help us see if the model is learning well or just memorizing training data.

Confusion Matrix Example

      | Predicted Positive | Predicted Negative |
      |--------------------|--------------------|
      | True Positive (TP)  | False Negative (FN) |
      | False Positive (FP) | True Negative (TN)  |

      Example:
      TP = 40, FP = 10
      FN = 5,  TN = 45
      Total samples = 100
    

This matrix helps calculate precision, recall, and accuracy during validation.

Precision vs Recall Tradeoff in Validation

During validation, precision and recall help us understand different errors:

• Precision = TP / (TP + FP): How many predicted positives are actually correct?
• Recall = TP / (TP + FN): How many actual positives did the model find?

For example, in a spam filter, high precision means fewer good emails marked as spam (important to avoid annoying users). High recall means catching most spam emails. Depending on the task, we choose which metric to prioritize during validation.

Good vs Bad Metric Values in Validation Loop

Good validation metrics:

• Validation loss steadily decreases or stays low.
• Validation accuracy is close to training accuracy (no big gap).
• Precision and recall are balanced and high for the task.

Bad validation metrics:

• Validation loss stops improving or increases (sign of overfitting).
• Validation accuracy much lower than training accuracy.
• Very low precision or recall indicating poor predictions.

Common Pitfalls in Validation Metrics

• Accuracy paradox: High accuracy can be misleading if classes are imbalanced.
• Data leakage: Validation data accidentally used in training leads to overly optimistic metrics.
• Overfitting indicators: Validation loss increasing while training loss decreases.
• Ignoring metric choice: Using accuracy alone when precision or recall matters more.

Self Check

Your model has 98% accuracy but only 12% recall on fraud cases during validation. Is it good for production? Why or why not?

Answer: No, it is not good. The low recall means the model misses most fraud cases, which is dangerous. Even with high accuracy, the model fails to catch fraud effectively. For fraud detection, high recall is critical to catch as many frauds as possible.

Key Result

Validation loss and accuracy show if the model generalizes well; precision and recall guide error tradeoffs.