Model comparison in Computer Vision - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is the main goal of model comparison in machine learning?

The main goal is to find which model performs best on a given task by comparing their accuracy, speed, and other metrics.

beginner

Name two common metrics used to compare computer vision models.

Accuracy and inference time are common metrics. Accuracy measures how often the model predicts correctly, and inference time measures how fast the model makes predictions.

intermediate

Why is it important to compare models on a validation set rather than the training set?

Because the validation set shows how well the model generalizes to new data, while the training set only shows how well it learned the examples it saw.

intermediate

What does it mean if one model has higher accuracy but slower inference time than another?

It means the first model is more accurate but takes longer to make predictions. Choosing between them depends on whether accuracy or speed is more important for the task.

beginner

How can visualizing model predictions help in model comparison?

Visualizing predictions helps spot where models make mistakes or succeed, giving insight beyond numbers and helping choose the best model for real-world use.

Which metric measures how fast a computer vision model makes predictions?

A Accuracy

B Inference time

C Loss

D Precision

Inference time measures the speed of the model's predictions.

Why should models be compared on a validation set instead of the training set?

A Training set is easier to use

B Training set is larger

C Validation set has fewer labels

D Validation set shows generalization to new data

Validation set tests how well the model works on unseen data.

If Model A is more accurate but slower than Model B, what should you consider?

A Always choose the faster model

B Ignore accuracy if speed is slow

C Whether accuracy or speed matters more for your task

D Use both models together

Choosing depends on the task's needs for accuracy versus speed.

Which of these is NOT a typical metric for model comparison?

A Model color

B Inference time

C Accuracy

D Precision

Model color is irrelevant for performance comparison.

How can visualizing model outputs help in comparison?

A Shows where models succeed or fail

B Increases model accuracy

C Reduces inference time

D Changes model architecture

Visualization helps understand model behavior beyond numbers.

Explain why comparing models using multiple metrics is important in computer vision.

Think about what matters more: speed or correctness.

Describe how you would decide between two models where one is faster but less accurate, and the other is slower but more accurate.

Consider the situation where the model will be used.

Practice

1. What is the main reason to compare different computer vision models on the same dataset?

easy

A. To find which model performs best for the task

B. To make the code run faster

C. To use more memory

D. To increase the dataset size

Solution

1. Step 1: Understand the purpose of model comparison

   Model comparison is done to evaluate which model gives better results on the same data.

2. Step 2: Identify the goal of comparing models

   The goal is to pick the best model for the task, not to affect code speed or data size.

3. Final Answer:

   To find which model performs best for the task -> Option A

4. Quick Check:

   Model comparison = find best model [OK]

Hint: Compare models by their results on the same data [OK]

Common Mistakes:

• Thinking comparison changes dataset size
• Confusing speed with model quality
• Assuming more memory means better model

2. Which of the following code snippets correctly compares two models' accuracy on the same test data in Python?

easy

A. acc1 = model1.fit(X_test, y_test)
acc2 = model2.fit(X_test, y_test)

B. acc1 = model1.evaluate(X_test, y_test)[1]
acc2 = model2.evaluate(X_test, y_test)[1]

C. acc1 = model1.predict(X_test)
acc2 = model2.predict(X_test)

D. acc1 = model1.score(X_train)
acc2 = model2.score(X_train)

Solution

1. Step 1: Identify correct method to get accuracy

   Using evaluate on test data returns loss and accuracy; index 1 is accuracy.

2. Step 2: Check other options for correctness

   fit trains, not evaluates; predict gives predictions, not accuracy; score needs both data and labels.

3. Final Answer:

   acc1 = model1.evaluate(X_test, y_test)[1] acc2 = model2.evaluate(X_test, y_test)[1] -> Option B

4. Quick Check:

   Use evaluate() for accuracy [OK]

Hint: Use evaluate() on test data to get accuracy [OK]

Common Mistakes:

• Using fit() instead of evaluate() for accuracy
• Using predict() output as accuracy
• Evaluating on training data instead of test data

3. Given the code below, what will be printed?

acc1 = 0.85
acc2 = 0.90
if acc1 > acc2:
    print('Model 1 is better')
else:
    print('Model 2 is better')

medium

A. Model 1 is better

B. Error: comparison not possible

C. Model 2 is better

D. No output

Solution

1. Step 1: Compare accuracy values

   acc1 is 0.85 and acc2 is 0.90, so acc1 < acc2.

2. Step 2: Follow the if-else logic

   Since acc1 > acc2 is false, the else block runs printing 'Model 2 is better'.

3. Final Answer:

   Model 2 is better -> Option C

4. Quick Check:

   0.85 < 0.90 so Model 2 wins [OK]

Hint: Compare accuracy numbers directly [OK]

Common Mistakes:

• Confusing greater than with less than
• Expecting error from simple comparison
• Ignoring else block output

4. You have two models but the code below gives an error. What is the problem?

acc1 = model1.evaluate(X_test, y_test)
acc2 = model2.evaluate(X_test, y_test)
if acc1 > acc2:
    print('Model 1 better')
else:
    print('Model 2 better')

medium

A. evaluate() returns a tuple, so direct comparison fails

B. X_test and y_test are swapped

C. Missing parentheses in print statements

D. Models are not trained yet

Solution

1. Step 1: Understand evaluate() output

   evaluate() returns a tuple (loss, accuracy), not a single number.

2. Step 2: Identify why comparison fails

   Comparing tuples directly with > causes error or unexpected behavior.

3. Final Answer:

   evaluate() returns a tuple, so direct comparison fails -> Option A

4. Quick Check:

   Compare accuracy values, not tuples [OK]

Hint: Extract accuracy from evaluate() tuple before comparing [OK]

Common Mistakes:

• Comparing full evaluate() output tuples
• Swapping test data inputs
• Assuming print syntax error

5. You want to compare three models on accuracy and speed. Which approach best helps you pick the best model?

hard

A. Use the model with smallest file size regardless of accuracy

B. Pick the model with highest accuracy only, ignoring speed

C. Choose the model with fastest training time only

D. Train all models, record accuracy and inference time, then choose the best trade-off

Solution

1. Step 1: Understand multiple criteria comparison

   Comparing models on both accuracy and speed requires measuring both metrics.

2. Step 2: Choose approach balancing accuracy and speed

   Recording accuracy and inference time helps find the best trade-off for your needs.

3. Final Answer:

   Train all models, record accuracy and inference time, then choose the best trade-off -> Option D

4. Quick Check:

   Balance accuracy and speed for best model [OK]

Hint: Measure both accuracy and speed, then compare [OK]

Common Mistakes:

• Ignoring speed when accuracy matters
• Choosing fastest training but poor accuracy
• Selecting model by size alone