Data augmentation in pipeline in TensorFlow - Model Metrics & Evaluation

Metrics & Evaluation - Data augmentation in pipeline

Which metric matters for Data Augmentation in Pipeline and WHY

Data augmentation helps the model see more varied examples by changing training data slightly. This usually improves validation accuracy and generalization. So, accuracy on unseen data is key to check if augmentation helps.

Also, watch loss during training and validation. If loss on validation decreases and accuracy increases, augmentation is working well.

Confusion Matrix Example

Imagine a model classifying images into cats and dogs. After augmentation, the confusion matrix on validation might look like this:

      | Predicted Cat | Predicted Dog |
      |---------------|---------------|
      | True Cat: 45  | 5             |
      | True Dog: 7   | 43            |
    

Here, total samples = 45 + 5 + 7 + 43 = 100.

Precision for Cat = TP / (TP + FP) = 45 / (45 + 7) = 0.865

Recall for Cat = TP / (TP + FN) = 45 / (45 + 5) = 0.9

Precision vs Recall Tradeoff with Data Augmentation

Data augmentation can help balance precision and recall by making the model robust to variations.

For example, in medical image classification, high recall is critical to catch all positive cases. Augmentation can help the model recognize more varied positive examples, improving recall.

In contrast, for spam detection, high precision is important to avoid marking good emails as spam. Augmentation should be done carefully to not confuse the model.

Good vs Bad Metric Values for Data Augmentation

Good: Validation accuracy improves or stays stable, validation loss decreases, precision and recall both improve or remain balanced.

Bad: Validation accuracy drops, validation loss increases, or precision and recall become very unbalanced (e.g., very high precision but very low recall).

Also watch for overfitting signs: training accuracy very high but validation accuracy low.

Common Pitfalls in Metrics with Data Augmentation

• Accuracy Paradox: Accuracy might look good if data is imbalanced. Always check precision and recall.
• Data Leakage: Augmenting validation or test data leaks training info and inflates metrics.
• Overfitting: Augmentation that is too weak or too strong can cause overfitting or underfitting, seen in diverging training and validation metrics.
• Ignoring Validation Metrics: Only looking at training metrics can mislead about real performance.

Self Check

Your model with data augmentation has 98% accuracy but 12% recall on the positive class (e.g., fraud). Is it good for production?

Answer: No. Despite high accuracy, the model misses most positive cases (low recall). For fraud detection, catching fraud (high recall) is critical. This model would fail in real use.

Key Result

Data augmentation improves model generalization, best measured by validation accuracy and balanced precision-recall.