Model comparison in Computer Vision

Introduction

Model comparison helps us find which model works best for our task by checking their results side by side.

When you have two or more models and want to pick the best one for recognizing images.

When you want to see if a new model improves accuracy compared to an old one.

When you want to check how fast different models run on your device.

When you want to balance between model size and performance for deployment.

When you want to understand which model makes fewer mistakes on your data.

Syntax

Computer Vision

1. Train or load multiple models.
2. Evaluate each model on the same test data.
3. Compare metrics like accuracy, loss, or speed.
4. Choose the model with the best balance for your needs.

Use the same test data for fair comparison.

Metrics can include accuracy, precision, recall, or inference time.

Examples

Compare accuracy scores of two models on the same test set.

Computer Vision

# Evaluate two models on test data
acc_model1 = model1.evaluate(test_images, test_labels, verbose=0)[1]
acc_model2 = model2.evaluate(test_images, test_labels, verbose=0)[1]
print(f"Model1 accuracy: {acc_model1}")
print(f"Model2 accuracy: {acc_model2}")

Measure how fast each model makes predictions on a few images.

Computer Vision

# Compare inference time
import time
start = time.time()
model1.predict(test_images[:10])
print(f"Model1 time: {time.time() - start} seconds")
start = time.time()
model2.predict(test_images[:10])
print(f"Model2 time: {time.time() - start} seconds")

Sample Model

This code trains two different CNN models on the MNIST dataset and compares their accuracy on the test set.

Computer Vision

import tensorflow as tf
from tensorflow.keras import layers, models

# Load sample data
(train_images, train_labels), (test_images, test_labels) = tf.keras.datasets.mnist.load_data()
train_images = train_images / 255.0
test_images = test_images / 255.0

# Model 1: Simple CNN
model1 = models.Sequential([
    layers.Reshape((28,28,1), input_shape=(28,28)),
    layers.Conv2D(16, 3, activation='relu'),
    layers.MaxPooling2D(),
    layers.Flatten(),
    layers.Dense(10, activation='softmax')
])
model1.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])
model1.fit(train_images, train_labels, epochs=2, verbose=0)

# Model 2: Deeper CNN
model2 = models.Sequential([
    layers.Reshape((28,28,1), input_shape=(28,28)),
    layers.Conv2D(32, 3, activation='relu'),
    layers.MaxPooling2D(),
    layers.Conv2D(64, 3, activation='relu'),
    layers.MaxPooling2D(),
    layers.Flatten(),
    layers.Dense(10, activation='softmax')
])
model2.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])
model2.fit(train_images, train_labels, epochs=2, verbose=0)

# Evaluate both models
loss1, acc1 = model1.evaluate(test_images, test_labels, verbose=0)
loss2, acc2 = model2.evaluate(test_images, test_labels, verbose=0)

print(f"Model 1 accuracy: {acc1:.4f}")
print(f"Model 2 accuracy: {acc2:.4f}")

Important Notes

Training for more epochs usually improves accuracy but takes longer.

A deeper model may perform better but needs more computing power.

Always use the same test data to compare models fairly.

Summary

Model comparison helps pick the best model by checking their results on the same data.

Compare models using accuracy, loss, or speed depending on your goal.

Simple code can train and evaluate multiple models to see which works best.

Practice

1. What is the main reason to compare different computer vision models on the same dataset?

easy

A. To find which model performs best for the task

B. To make the code run faster

C. To use more memory

D. To increase the dataset size

Solution

1. Step 1: Understand the purpose of model comparison

   Model comparison is done to evaluate which model gives better results on the same data.

2. Step 2: Identify the goal of comparing models

   The goal is to pick the best model for the task, not to affect code speed or data size.

3. Final Answer:

   To find which model performs best for the task -> Option A

4. Quick Check:

   Model comparison = find best model [OK]

Hint: Compare models by their results on the same data [OK]

Common Mistakes:

• Thinking comparison changes dataset size
• Confusing speed with model quality
• Assuming more memory means better model

2. Which of the following code snippets correctly compares two models' accuracy on the same test data in Python?

easy

A. acc1 = model1.fit(X_test, y_test)
acc2 = model2.fit(X_test, y_test)

B. acc1 = model1.evaluate(X_test, y_test)[1]
acc2 = model2.evaluate(X_test, y_test)[1]

C. acc1 = model1.predict(X_test)
acc2 = model2.predict(X_test)

D. acc1 = model1.score(X_train)
acc2 = model2.score(X_train)

Solution

1. Step 1: Identify correct method to get accuracy

   Using evaluate on test data returns loss and accuracy; index 1 is accuracy.

2. Step 2: Check other options for correctness

   fit trains, not evaluates; predict gives predictions, not accuracy; score needs both data and labels.

3. Final Answer:

   acc1 = model1.evaluate(X_test, y_test)[1] acc2 = model2.evaluate(X_test, y_test)[1] -> Option B

4. Quick Check:

   Use evaluate() for accuracy [OK]

Hint: Use evaluate() on test data to get accuracy [OK]

Common Mistakes:

• Using fit() instead of evaluate() for accuracy
• Using predict() output as accuracy
• Evaluating on training data instead of test data

3. Given the code below, what will be printed?

acc1 = 0.85
acc2 = 0.90
if acc1 > acc2:
    print('Model 1 is better')
else:
    print('Model 2 is better')

medium

A. Model 1 is better

B. Error: comparison not possible

C. Model 2 is better

D. No output

Solution

1. Step 1: Compare accuracy values

   acc1 is 0.85 and acc2 is 0.90, so acc1 < acc2.

2. Step 2: Follow the if-else logic

   Since acc1 > acc2 is false, the else block runs printing 'Model 2 is better'.

3. Final Answer:

   Model 2 is better -> Option C

4. Quick Check:

   0.85 < 0.90 so Model 2 wins [OK]

Hint: Compare accuracy numbers directly [OK]

Common Mistakes:

• Confusing greater than with less than
• Expecting error from simple comparison
• Ignoring else block output

4. You have two models but the code below gives an error. What is the problem?

acc1 = model1.evaluate(X_test, y_test)
acc2 = model2.evaluate(X_test, y_test)
if acc1 > acc2:
    print('Model 1 better')
else:
    print('Model 2 better')

medium

A. evaluate() returns a tuple, so direct comparison fails

B. X_test and y_test are swapped

C. Missing parentheses in print statements

D. Models are not trained yet

Solution

1. Step 1: Understand evaluate() output

   evaluate() returns a tuple (loss, accuracy), not a single number.

2. Step 2: Identify why comparison fails

   Comparing tuples directly with > causes error or unexpected behavior.

3. Final Answer:

   evaluate() returns a tuple, so direct comparison fails -> Option A

4. Quick Check:

   Compare accuracy values, not tuples [OK]

Hint: Extract accuracy from evaluate() tuple before comparing [OK]

Common Mistakes:

• Comparing full evaluate() output tuples
• Swapping test data inputs
• Assuming print syntax error

5. You want to compare three models on accuracy and speed. Which approach best helps you pick the best model?

hard

A. Use the model with smallest file size regardless of accuracy

B. Pick the model with highest accuracy only, ignoring speed

C. Choose the model with fastest training time only

D. Train all models, record accuracy and inference time, then choose the best trade-off

Solution

1. Step 1: Understand multiple criteria comparison

   Comparing models on both accuracy and speed requires measuring both metrics.

2. Step 2: Choose approach balancing accuracy and speed

   Recording accuracy and inference time helps find the best trade-off for your needs.

3. Final Answer:

   Train all models, record accuracy and inference time, then choose the best trade-off -> Option D

4. Quick Check:

   Balance accuracy and speed for best model [OK]

Hint: Measure both accuracy and speed, then compare [OK]

Common Mistakes:

• Ignoring speed when accuracy matters
• Choosing fastest training but poor accuracy
• Selecting model by size alone