Epoch-based training in PyTorch - Model Metrics & Evaluation

Metrics & Evaluation - Epoch-based training

Which metric matters for Epoch-based training and WHY

When training a model in epochs, the key metrics to watch are training loss and validation loss. Loss tells us how well the model is learning to predict the right answers. We want the loss to go down as epochs increase. Also, validation accuracy helps us see if the model is improving on new data it hasn't seen before. These metrics show if the model is learning well or just memorizing the training data.

Confusion matrix example after epoch training

Suppose after training for 5 epochs, the model predicts on a test set of 100 samples. The confusion matrix might look like this:

      | Predicted Positive | Predicted Negative |
      |--------------------|--------------------|
      | True Positive (TP): 40 | False Positive (FP): 5 |
      | False Negative (FN): 10 | True Negative (TN): 45 |
    

From this, we can calculate precision, recall, and accuracy to understand model performance after training.

Precision vs Recall tradeoff during epoch training

As epochs increase, the model may become better at finding positive cases (higher recall) but might also make more mistakes (lower precision). For example, in a spam filter, if the model catches almost all spam (high recall) but marks many good emails as spam (low precision), users get annoyed. So, during epoch training, we watch these metrics to find a balance that fits our goal.

What good vs bad metric values look like after epoch training

Good: Training and validation loss both decrease and stabilize. Validation accuracy improves and stays close to training accuracy. Precision and recall are balanced according to the task needs.

Bad: Training loss keeps going down but validation loss goes up (overfitting). Large gap between training and validation accuracy. Precision or recall very low, meaning the model misses many cases or makes many mistakes.

Common pitfalls in epoch-based training metrics

• Accuracy paradox: High accuracy can be misleading if classes are imbalanced.
• Overfitting: Training loss decreases but validation loss increases.
• Data leakage: Validation data accidentally used in training can give false good metrics.
• Early stopping ignored: Not stopping training when validation loss stops improving wastes time and causes overfitting.

Self-check question

Your model has 98% accuracy but only 12% recall on fraud cases after 10 epochs. Is it good for production? Why or why not?

Answer: No, it is not good. The model misses 88% of fraud cases (low recall), which is dangerous. High accuracy is misleading because fraud cases are rare. We need to improve recall to catch more fraud.

Key Result

In epoch-based training, watching loss and validation accuracy over epochs helps detect learning progress and overfitting.