Image datasets (CIFAR-10, ImageNet) in Computer Vision - Model Metrics & Evaluation

Metrics & Evaluation - Image datasets (CIFAR-10, ImageNet)

Which metric matters for Image datasets (CIFAR-10, ImageNet) and WHY

For image classification tasks using datasets like CIFAR-10 and ImageNet, accuracy is the most common metric. This is because these datasets have balanced classes and the goal is to correctly identify the image category.

However, when classes are imbalanced or some mistakes are more costly, precision, recall, and F1 score become important. For example, if missing a rare class is bad, recall matters more.

In large datasets like ImageNet, top-1 and top-5 accuracy are used. Top-1 accuracy checks if the model's best guess is correct. Top-5 accuracy checks if the correct label is among the model's five best guesses, which is useful when many classes look similar.

Confusion matrix example for CIFAR-10 (10 classes)

      | Predicted Class -->
      | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 |
    -----------------------------------------
    0 |50 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
    1 | 1 |48 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 |
    2 | 0 | 0 |45 | 2 | 0 | 0 | 0 | 0 | 3 | 0 |
    3 | 0 | 0 | 1 |47 | 0 | 0 | 0 | 0 | 2 | 0 |
    4 | 0 | 0 | 0 | 0 |49 | 0 | 0 | 0 | 1 | 0 |
    5 | 0 | 0 | 0 | 0 | 0 |50 | 0 | 0 | 0 | 0 |
    6 | 0 | 0 | 0 | 0 | 0 | 0 |50 | 0 | 0 | 0 |
    7 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |50 | 0 | 0 |
    8 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 |49 | 0 |
    9 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |50 |
    

This matrix shows mostly correct predictions on the diagonal. Off-diagonal numbers are mistakes.

Precision vs Recall tradeoff with image datasets

Imagine a model classifying animals in images. If it labels many images as "cat" (high recall), it might also wrongly label dogs as cats (low precision).

If you want to avoid wrong labels (high precision), the model might miss some cats (low recall).

For CIFAR-10 or ImageNet, usually balanced accuracy is enough. But if you focus on a rare class (like "frog"), you might want to tune for higher recall to catch all frogs, even if some mistakes happen.

What "good" vs "bad" metric values look like for CIFAR-10 and ImageNet

CIFAR-10: Good accuracy is above 85% for simple models, above 95% for strong models. Bad accuracy is below 70%, meaning many mistakes.

ImageNet: Good top-1 accuracy is above 75%, top-5 accuracy above 90%. Bad models have top-1 accuracy below 50%, meaning they guess wrong most of the time.

Precision and recall should be close to accuracy for balanced classes. Large gaps may mean the model favors some classes over others.

Common pitfalls when evaluating image dataset models

• Accuracy paradox: High accuracy can hide poor performance on rare classes.
• Data leakage: Using test images in training inflates metrics falsely.
• Overfitting: Very high training accuracy but low test accuracy means the model memorizes images, not generalizes.
• Ignoring top-5 accuracy: For ImageNet, only checking top-1 can miss useful model behavior.
• Class imbalance: Not checking per-class metrics can hide poor results on some categories.

Self-check question

Your model on ImageNet has 98% accuracy but only 12% recall on a rare class like "otter." Is it good for production?

Answer: No. While overall accuracy is high, the model misses most otters (low recall). This means it fails to find many otters, which could be critical depending on the use case. You should improve recall for that class before production.

Key Result

Accuracy is key for balanced image datasets; top-1 and top-5 accuracy are essential for ImageNet; precision and recall matter for rare classes.