Small dataset strategies in Computer Vision - Model Metrics & Evaluation

Metrics & Evaluation - Small dataset strategies

Which metric matters for Small Dataset Strategies and WHY

When working with small datasets in computer vision, accuracy, precision, and recall are important to check if the model learns well without overfitting. Overfitting means the model memorizes the small data but fails on new images.

We also look at validation loss and training loss to see if the model generalizes. If validation loss is much higher than training loss, the model is overfitting.

Metrics like F1 score help balance precision and recall, especially if classes are imbalanced.

Confusion Matrix Example

      Actual \ Predicted | Cat | Dog
      -------------------|-----|-----
      Cat                |  8  |  2  
      Dog                |  1  |  9  

      Total samples = 20
      TP (Cat) = 8, FP (Cat) = 1, FN (Cat) = 2, TN (Cat) = 9
    

This matrix helps calculate precision and recall for each class to understand model errors.

Precision vs Recall Tradeoff with Small Datasets

With small data, models may miss some objects (low recall) or wrongly detect objects (low precision).

For example, in medical image detection, high recall is critical to catch all cases, even if some false alarms happen.

In contrast, for a photo app that tags pets, high precision is better to avoid wrong tags.

Balancing precision and recall with F1 score helps decide the best model for your small dataset.

Good vs Bad Metric Values for Small Dataset Models

Good: Validation accuracy close to training accuracy, precision and recall above 80%, and F1 score balanced.

Bad: Very high training accuracy (e.g., 99%) but low validation accuracy (e.g., 60%), showing overfitting.

Also, very low recall (e.g., 30%) means the model misses many true objects.

Common Pitfalls in Metrics with Small Datasets

• Accuracy paradox: High accuracy can be misleading if classes are imbalanced.
• Data leakage: Accidentally using test images in training inflates metrics falsely.
• Overfitting: Model memorizes training images but fails on new ones, seen by big gap between training and validation metrics.
• Small sample size: Metrics can vary a lot due to few examples, so use cross-validation or data augmentation.

Self-Check Question

Your model trained on 100 images has 98% accuracy but only 12% recall on the rare class. Is it good for production?

Answer: No. The model misses most of the rare class cases (low recall), so it is not reliable despite high accuracy.

Key Result

For small datasets, balanced precision and recall with close training and validation accuracy indicate a good model without overfitting.

Practice

1. Which of the following is a common strategy to improve model performance when you have a small image dataset?

easy

A. Train a deep model from scratch without any pre-trained weights

B. Use data augmentation to create more training images

C. Ignore validation to use all data for training

D. Reduce image resolution to save memory only

Solution

1. Step 1: Understand small dataset challenges

   Small datasets often cause models to overfit and perform poorly on new data.

2. Step 2: Identify effective strategies

   Data augmentation creates new images by modifying existing ones, increasing data variety and helping the model generalize better.

3. Final Answer:

   Use data augmentation to create more training images -> Option B

4. Quick Check:

   Data augmentation = More data variety [OK]

Hint: More data variety helps small datasets [OK]

Common Mistakes:

• Training from scratch causes overfitting
• Ignoring validation hides model issues
• Reducing resolution alone doesn't add data

2. Which code snippet correctly applies data augmentation using the Python library torchvision.transforms?

easy

A. transforms.Compose([transforms.RandomHorizontalFlip(), transforms.ToTensor()])

B. transforms.RandomCrop(32, 32)

C. transforms.ToTensor(), transforms.Normalize()

D. transforms.Resize(256)

Solution

1. Step 1: Recognize data augmentation syntax

   Data augmentation requires combining multiple transforms, usually with Compose.

2. Step 2: Check which option uses Compose with augmentation

   transforms.Compose([transforms.RandomHorizontalFlip(), transforms.ToTensor()]) uses Compose with RandomHorizontalFlip (augmentation) and ToTensor (conversion), which is correct.

3. Final Answer:

   transforms.Compose([transforms.RandomHorizontalFlip(), transforms.ToTensor()]) -> Option A

4. Quick Check:

   Compose + augmentation = transforms.Compose([transforms.RandomHorizontalFlip(), transforms.ToTensor()]) [OK]

Hint: Use Compose to combine augmentations [OK]

Common Mistakes:

• Using single transform without Compose
• Missing ToTensor conversion
• Using only resizing without augmentation

3. Consider this Python code using transfer learning with PyTorch:

import torchvision.models as models
model = models.resnet18(pretrained=True)
for param in model.parameters():
    param.requires_grad = False
model.fc = torch.nn.Linear(512, 2)

What does this code do?

medium

A. Trains all layers of ResNet18 from scratch

B. Unfreezes all layers for fine-tuning

C. Freezes all layers except the last fully connected layer

D. Removes the last layer without replacement

Solution

1. Step 1: Analyze parameter freezing

   The loop sets requires_grad=False for all parameters, freezing them during training.

2. Step 2: Check the last layer replacement

   The last fully connected layer (fc) is replaced with a new Linear layer, which by default has requires_grad=True.

3. Final Answer:

   Freezes all layers except the last fully connected layer -> Option C

4. Quick Check:

   Freeze all but last layer = Freezes all layers except the last fully connected layer [OK]

Hint: Freeze parameters, then replace last layer [OK]

Common Mistakes:

• Assuming all layers are trainable
• Not noticing last layer replacement
• Confusing freezing with unfreezing

4. You wrote this code to augment images but get an error:

transform = transforms.Compose([
    transforms.RandomRotation(30),
    transforms.ToTensor
])

What is the error and how to fix it?

medium

A. Transforms must be applied outside Compose

B. RandomRotation requires degrees as a tuple, fix by using (0,30)

C. Compose should be replaced by Sequential

D. Missing parentheses after ToTensor; fix by using transforms.ToTensor()

Solution

1. Step 1: Identify the error in ToTensor usage

   transforms.ToTensor is a class, missing parentheses means it's not called, causing an error.

2. Step 2: Correct the syntax

   Add parentheses to call ToTensor: transforms.ToTensor()

3. Final Answer:

   Missing parentheses after ToTensor; fix by using transforms.ToTensor() -> Option D

4. Quick Check:

   Call ToTensor() as function [OK]

Hint: Call transform classes with () [OK]

Common Mistakes:

• Forgetting parentheses on transform classes
• Misusing Compose with wrong functions
• Incorrect argument types for RandomRotation

5. You have only 100 labeled images for a classification task. Which combined approach best improves model accuracy?

hard

A. Use transfer learning with a pre-trained model and apply data augmentation

B. Train a deep CNN from scratch with no augmentation

C. Use only data augmentation without pre-trained weights

D. Increase batch size to 512 and train for fewer epochs

Solution

1. Step 1: Understand small dataset limits

   With only 100 images, training deep models from scratch risks overfitting and poor generalization.

2. Step 2: Combine transfer learning and augmentation

   Transfer learning uses knowledge from large datasets, and augmentation increases data variety, both improving accuracy.

3. Final Answer:

   Use transfer learning with a pre-trained model and apply data augmentation -> Option A

4. Quick Check:

   Transfer learning + augmentation = Best for small data [OK]

Hint: Combine pre-trained models with augmentation [OK]

Common Mistakes:

• Training from scratch with little data
• Relying on augmentation alone
• Using too large batch size causing poor learning