Data augmentation with transforms in PyTorch - Model Metrics & Evaluation

Metrics & Evaluation - Data augmentation with transforms

Which metric matters for Data Augmentation and WHY

Data augmentation helps models learn better by showing more varied examples. The key metric to watch is validation accuracy or validation loss. This shows if the model is truly learning to generalize to new data, not just memorizing training data.

We focus on validation metrics because augmentation aims to reduce overfitting. If validation accuracy improves or validation loss decreases after augmentation, it means the transforms helped the model learn better.

Confusion Matrix Example

For classification tasks, a confusion matrix shows how well the model predicts each class after augmentation.

      Actual \ Predicted | Cat | Dog | Rabbit
      -------------------|-----|-----|-------
      Cat                | 45  | 3   | 2     
      Dog                | 4   | 43  | 3     
      Rabbit             | 1   | 2   | 47    
    

This matrix sums to 150 samples. From it, we calculate precision, recall, and F1 for each class to see if augmentation improved class-wise predictions.

Precision vs Recall Tradeoff with Augmentation

Augmentation can affect precision and recall differently. For example, if augmentation adds noise, the model might become more cautious, increasing precision but lowering recall.

Imagine a dog detector: high recall means catching most dogs, but low precision means some cats are wrongly labeled as dogs. Augmentation should balance this by improving recall without hurting precision too much.

Good vs Bad Metric Values for Augmentation

Good: Validation accuracy increases by a few points, validation loss decreases, and class-wise precision and recall improve or stay stable.

Bad: Validation accuracy stays the same or drops, validation loss increases, or precision/recall drop significantly, indicating augmentation harmed learning.

Common Pitfalls in Metrics with Augmentation

• Accuracy paradox: Accuracy may look good if classes are imbalanced, hiding poor performance on rare classes.
• Data leakage: Augmentation applied before splitting data can leak test info into training, inflating metrics.
• Overfitting indicators: If training accuracy is very high but validation accuracy is low, augmentation might be insufficient or wrongly applied.

Self Check

Your model has 98% accuracy but only 12% recall on fraud cases after augmentation. Is it good?

Answer: No. Despite high accuracy, the model misses most fraud cases (low recall). For fraud detection, recall is critical to catch fraud. Augmentation or model needs improvement to raise recall.

Key Result

Validation accuracy and recall are key to evaluate if data augmentation helps the model generalize better.