Experiment - Top-K accuracy

Problem:You are training a computer vision model to classify images into 10 categories. The current model achieves 85% accuracy on the training set but only 70% accuracy on the validation set. You want to better understand how often the correct label is among the top 3 predictions, not just the top 1.

Current Metrics:Training accuracy: 85%, Validation accuracy: 70%, Top-1 accuracy only

Issue:The model's top-1 accuracy on validation is low, and you want to evaluate top-K accuracy (top-3) to see if the model predicts the correct label within its top 3 guesses.

Your Task

Calculate and report the top-3 accuracy on the validation set to better understand model performance beyond top-1 accuracy.

Use the existing trained model without retraining.

Do not change the model architecture.

Use only the validation dataset for this evaluation.

Hint 1

Hint 2

Hint 3

Solution

Computer Vision

import tensorflow as tf
from tensorflow.keras.datasets import cifar10
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D, MaxPooling2D, Flatten, Dense
from tensorflow.keras.utils import to_categorical

# Load data
(x_train, y_train), (x_test, y_test) = cifar10.load_data()

# Normalize images
x_train, x_test = x_train / 255.0, x_test / 255.0

# Convert labels to categorical
num_classes = 10
y_train_cat = to_categorical(y_train, num_classes)
y_test_cat = to_categorical(y_test, num_classes)

# Define a simple CNN model
model = Sequential([
    Conv2D(32, (3,3), activation='relu', input_shape=(32,32,3)),
    MaxPooling2D((2,2)),
    Conv2D(64, (3,3), activation='relu'),
    MaxPooling2D((2,2)),
    Flatten(),
    Dense(64, activation='relu'),
    Dense(num_classes, activation='softmax')
])

model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

# Train model briefly for demonstration
model.fit(x_train, y_train_cat, epochs=3, batch_size=64, validation_split=0.1, verbose=0)

# Evaluate top-1 accuracy
loss, top1_acc = model.evaluate(x_test, y_test_cat, verbose=0)

# Calculate top-3 accuracy
# Use tf.keras.metrics.TopKCategoricalAccuracy
top3_acc_metric = tf.keras.metrics.TopKCategoricalAccuracy(k=3)
top3_acc_metric.update_state(y_test_cat, model.predict(x_test, verbose=0))
top3_acc = top3_acc_metric.result().numpy()

print(f"Top-1 accuracy on test set: {top1_acc*100:.2f}%")
print(f"Top-3 accuracy on test set: {top3_acc*100:.2f}%")

Added calculation of top-3 accuracy using TensorFlow's TopKCategoricalAccuracy metric.

Kept the original model and dataset unchanged.

Reported both top-1 and top-3 accuracy for comparison.

Results Interpretation

Before: Only top-1 accuracy was reported at 70.5%.
After: Top-3 accuracy is 89.3%, showing the model predicts the correct label within its top 3 guesses much more often.

Top-K accuracy helps us understand model performance beyond just the single best guess. It is useful in applications where having the correct answer in the top few predictions is acceptable.

Bonus Experiment

Try calculating top-5 accuracy and compare it with top-1 and top-3 accuracies.

💡 Hint

Use the same TopKCategoricalAccuracy metric with k=5 and observe how accuracy improves as k increases.