Training loop structure in PyTorch - Model Metrics & Evaluation

Metrics & Evaluation - Training loop structure

Which metric matters for Training Loop Structure and WHY

When training a model, the key metrics to watch are training loss and validation loss. Loss tells us how far off the model's predictions are from the true answers. Lower loss means better learning.

Accuracy is also important if the task is classification. It shows the percentage of correct predictions.

We track these metrics each training step or epoch to see if the model is improving or stuck.

Confusion Matrix Example

Confusion Matrix (for classification):

          Predicted
          0    1
Actual 0 50   10
       1  5   35

TP = 35 (correctly predicted 1s)
FP = 10 (wrongly predicted 1s)
TN = 50 (correctly predicted 0s)
FN = 5  (missed 1s)

Total samples = 50 + 10 + 5 + 35 = 100
    

This matrix helps calculate precision, recall, and accuracy during training evaluation.

Precision vs Recall Tradeoff in Training

During training, improving one metric can lower another. For example:

• Precision: How many predicted positives are actually positive.
• Recall: How many actual positives were found.

If the model is too strict, it may have high precision but low recall (miss many positives). If too loose, high recall but low precision (many false alarms).

Choosing which to prioritize depends on the problem. Training loops should monitor both to balance them.

Good vs Bad Metric Values During Training

Good:

• Training and validation loss steadily decrease.
• Accuracy improves and stabilizes at a high value.
• Precision and recall both improve without big gaps.

Bad:

• Training loss decreases but validation loss increases (overfitting).
• Accuracy stays low or fluctuates wildly.
• Precision very high but recall very low, or vice versa.

Common Pitfalls in Training Loop Metrics

• Accuracy paradox: High accuracy can be misleading if classes are imbalanced.
• Data leakage: Validation data accidentally used in training inflates metrics.
• Overfitting: Training loss drops but validation loss rises, showing poor generalization.
• Ignoring loss curves: Not checking loss trends can miss training issues.

Self Check

Your model has 98% accuracy but 12% recall on fraud detection. Is it good?

No. The model misses 88% of fraud cases (low recall), which is dangerous. High accuracy is misleading because fraud is rare. You need to improve recall to catch more fraud.

Key Result

Training and validation loss trends are key to judge if the training loop is working well.