Small dataset strategies in Computer Vision - Model Metrics & Evaluation

Metrics & Evaluation - Small dataset strategies

Which metric matters for Small Dataset Strategies and WHY

When working with small datasets in computer vision, accuracy, precision, and recall are important to check if the model learns well without overfitting. Overfitting means the model memorizes the small data but fails on new images.

We also look at validation loss and training loss to see if the model generalizes. If validation loss is much higher than training loss, the model is overfitting.

Metrics like F1 score help balance precision and recall, especially if classes are imbalanced.

Confusion Matrix Example

      Actual \ Predicted | Cat | Dog
      -------------------|-----|-----
      Cat                |  8  |  2  
      Dog                |  1  |  9  

      Total samples = 20
      TP (Cat) = 8, FP (Cat) = 1, FN (Cat) = 2, TN (Cat) = 9
    

This matrix helps calculate precision and recall for each class to understand model errors.

Precision vs Recall Tradeoff with Small Datasets

With small data, models may miss some objects (low recall) or wrongly detect objects (low precision).

For example, in medical image detection, high recall is critical to catch all cases, even if some false alarms happen.

In contrast, for a photo app that tags pets, high precision is better to avoid wrong tags.

Balancing precision and recall with F1 score helps decide the best model for your small dataset.

Good vs Bad Metric Values for Small Dataset Models

Good: Validation accuracy close to training accuracy, precision and recall above 80%, and F1 score balanced.

Bad: Very high training accuracy (e.g., 99%) but low validation accuracy (e.g., 60%), showing overfitting.

Also, very low recall (e.g., 30%) means the model misses many true objects.

Common Pitfalls in Metrics with Small Datasets

• Accuracy paradox: High accuracy can be misleading if classes are imbalanced.
• Data leakage: Accidentally using test images in training inflates metrics falsely.
• Overfitting: Model memorizes training images but fails on new ones, seen by big gap between training and validation metrics.
• Small sample size: Metrics can vary a lot due to few examples, so use cross-validation or data augmentation.

Self-Check Question

Your model trained on 100 images has 98% accuracy but only 12% recall on the rare class. Is it good for production?

Answer: No. The model misses most of the rare class cases (low recall), so it is not reliable despite high accuracy.

Key Result

For small datasets, balanced precision and recall with close training and validation accuracy indicate a good model without overfitting.