Image datasets (CIFAR-10, ImageNet) in Computer Vision - Model Metrics & Evaluation

Metrics & Evaluation - Image datasets (CIFAR-10, ImageNet)

Which metric matters for Image datasets (CIFAR-10, ImageNet) and WHY

For image classification tasks using datasets like CIFAR-10 and ImageNet, accuracy is the most common metric. This is because these datasets have balanced classes and the goal is to correctly identify the image category.

However, when classes are imbalanced or some mistakes are more costly, precision, recall, and F1 score become important. For example, if missing a rare class is bad, recall matters more.

In large datasets like ImageNet, top-1 and top-5 accuracy are used. Top-1 accuracy checks if the model's best guess is correct. Top-5 accuracy checks if the correct label is among the model's five best guesses, which is useful when many classes look similar.

Confusion matrix example for CIFAR-10 (10 classes)

      | Predicted Class -->
      | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 |
    -----------------------------------------
    0 |50 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
    1 | 1 |48 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 |
    2 | 0 | 0 |45 | 2 | 0 | 0 | 0 | 0 | 3 | 0 |
    3 | 0 | 0 | 1 |47 | 0 | 0 | 0 | 0 | 2 | 0 |
    4 | 0 | 0 | 0 | 0 |49 | 0 | 0 | 0 | 1 | 0 |
    5 | 0 | 0 | 0 | 0 | 0 |50 | 0 | 0 | 0 | 0 |
    6 | 0 | 0 | 0 | 0 | 0 | 0 |50 | 0 | 0 | 0 |
    7 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |50 | 0 | 0 |
    8 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 |49 | 0 |
    9 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |50 |
    

This matrix shows mostly correct predictions on the diagonal. Off-diagonal numbers are mistakes.

Precision vs Recall tradeoff with image datasets

Imagine a model classifying animals in images. If it labels many images as "cat" (high recall), it might also wrongly label dogs as cats (low precision).

If you want to avoid wrong labels (high precision), the model might miss some cats (low recall).

For CIFAR-10 or ImageNet, usually balanced accuracy is enough. But if you focus on a rare class (like "frog"), you might want to tune for higher recall to catch all frogs, even if some mistakes happen.

What "good" vs "bad" metric values look like for CIFAR-10 and ImageNet

CIFAR-10: Good accuracy is above 85% for simple models, above 95% for strong models. Bad accuracy is below 70%, meaning many mistakes.

ImageNet: Good top-1 accuracy is above 75%, top-5 accuracy above 90%. Bad models have top-1 accuracy below 50%, meaning they guess wrong most of the time.

Precision and recall should be close to accuracy for balanced classes. Large gaps may mean the model favors some classes over others.

Common pitfalls when evaluating image dataset models

• Accuracy paradox: High accuracy can hide poor performance on rare classes.
• Data leakage: Using test images in training inflates metrics falsely.
• Overfitting: Very high training accuracy but low test accuracy means the model memorizes images, not generalizes.
• Ignoring top-5 accuracy: For ImageNet, only checking top-1 can miss useful model behavior.
• Class imbalance: Not checking per-class metrics can hide poor results on some categories.

Self-check question

Your model on ImageNet has 98% accuracy but only 12% recall on a rare class like "otter." Is it good for production?

Answer: No. While overall accuracy is high, the model misses most otters (low recall). This means it fails to find many otters, which could be critical depending on the use case. You should improve recall for that class before production.

Key Result

Accuracy is key for balanced image datasets; top-1 and top-5 accuracy are essential for ImageNet; precision and recall matter for rare classes.

Practice

1. Which of the following best describes the CIFAR-10 dataset?

easy

A. A small dataset with 10 classes of images, easy for beginners

B. A very large dataset with millions of images and thousands of classes

C. A dataset mainly used for text recognition tasks

D. A dataset containing only black and white images

Solution

1. Step 1: Understand CIFAR-10 size and classes

   CIFAR-10 contains 60,000 small images divided into 10 classes, making it manageable for beginners.

2. Step 2: Compare with other datasets

   ImageNet is much larger with many more classes, unlike CIFAR-10.

3. Final Answer:

   A small dataset with 10 classes of images, easy for beginners -> Option A

4. Quick Check:

   CIFAR-10 = small, 10 classes [OK]

Hint: Remember CIFAR-10 is small and simple for learning [OK]

Common Mistakes:

• Confusing CIFAR-10 with ImageNet size
• Thinking CIFAR-10 has many classes
• Assuming CIFAR-10 is for text data

2. Which Python code correctly loads the CIFAR-10 dataset using TensorFlow?

easy

A. import cifar10
train_images, train_labels = cifar10.load()

B. from tensorflow.keras.datasets import cifar10
(train_images, train_labels), (test_images, test_labels) = cifar10.load_data()

C. from tensorflow.data import cifar10
train, test = cifar10.load()

D. from keras.datasets import imagenet
train, test = imagenet.load_data()

Solution

1. Step 1: Identify correct import for CIFAR-10 in TensorFlow

   The correct import is from tensorflow.keras.datasets import cifar10.

2. Step 2: Check the loading function

   cifar10.load_data() returns training and testing sets as tuples.

3. Final Answer:

   from tensorflow.keras.datasets import cifar10 (train_images, train_labels), (test_images, test_labels) = cifar10.load_data() -> Option B

4. Quick Check:

   Correct import and load_data() method [OK]

Hint: Use tensorflow.keras.datasets for CIFAR-10 loading [OK]

Common Mistakes:

• Using wrong module names like tensorflow.data
• Trying to load ImageNet with CIFAR-10 code
• Missing the load_data() function call

3. What will be the shape of the training images array after loading CIFAR-10 with TensorFlow?

medium

A. (100000, 64, 64, 3)

B. (60000, 28, 28, 1)

C. (50000, 32, 32, 3)

D. (50000, 224, 224, 3)

Solution

1. Step 1: Recall CIFAR-10 image count and size

   CIFAR-10 has 50,000 training images, each 32x32 pixels with 3 color channels (RGB).

2. Step 2: Match shape format

   The shape is (number_of_images, height, width, channels) = (50000, 32, 32, 3).

3. Final Answer:

   (50000, 32, 32, 3) -> Option C

4. Quick Check:

   Training images shape = (50000, 32, 32, 3) [OK]

Hint: CIFAR-10 images are 32x32 RGB, 50k training samples [OK]

Common Mistakes:

• Confusing CIFAR-10 with MNIST image size
• Using ImageNet image dimensions
• Mixing training and test set sizes

4. You wrote this code to load ImageNet but get an error:

from tensorflow.keras.datasets import imagenet
(train_images, train_labels), (test_images, test_labels) = imagenet.load_data()

What is the main problem?

medium

A. ImageNet is not available in tensorflow.keras.datasets module

B. The load_data() function requires extra parameters

C. You must import ImageNet from tensorflow.data instead

D. ImageNet images are grayscale, so loading fails

Solution

1. Step 1: Check TensorFlow dataset availability

   TensorFlow's keras.datasets does not include ImageNet; it includes CIFAR-10, MNIST, etc.

2. Step 2: Understand ImageNet loading method

   ImageNet requires special handling or external libraries, not keras.datasets.

3. Final Answer:

   ImageNet is not available in tensorflow.keras.datasets module -> Option A

4. Quick Check:

   ImageNet not in keras.datasets [OK]

Hint: ImageNet needs special loading, not keras.datasets [OK]

Common Mistakes:

• Assuming ImageNet loads like CIFAR-10
• Trying to import from wrong TensorFlow submodules
• Believing ImageNet images are grayscale

5. You want to train a model to recognize 1000 different object categories. Which dataset is best suited for this task?

hard

A. CIFAR-10, because it has 10 classes and is easy to use

B. Fashion-MNIST, because it has clothing images

C. MNIST, because it has handwritten digits

D. ImageNet, because it has 1000 classes and many images per class

Solution

1. Step 1: Identify dataset class count

   CIFAR-10 has only 10 classes, MNIST and Fashion-MNIST have 10 classes each, ImageNet has 1000 classes.

2. Step 2: Match dataset to task

   For recognizing 1000 categories, ImageNet is the suitable dataset due to its size and diversity.

3. Final Answer:

   ImageNet, because it has 1000 classes and many images per class -> Option D

4. Quick Check:

   1000 classes need ImageNet [OK]

Hint: Use ImageNet for many classes, CIFAR-10 for few [OK]

Common Mistakes:

• Choosing CIFAR-10 for many classes
• Confusing MNIST with ImageNet
• Ignoring class count importance