Model comparison in Computer Vision - Model Metrics & Evaluation

Metrics & Evaluation - Model comparison

Which metric matters for Model Comparison and WHY

When comparing models, we want to know which one works best for our task. In computer vision, common metrics include accuracy, precision, recall, and F1 score. These metrics tell us how well the model predicts images correctly, especially when classes are unbalanced. For example, if we want to detect rare objects, recall is important to catch as many as possible. If we want to avoid false alarms, precision matters. Using these metrics helps us pick the model that fits our real-life needs.

Confusion Matrix Example

      Predicted
      | Cat | Dog |
    ---+-----+-----+
    Cat| 50  | 10  |
    Dog| 5   | 35  |

    True labels: 50 cats correctly predicted (TP for cat), 10 cats missed (FN for cat), 5 dogs wrongly predicted as cats (FP for cat), 35 dogs correctly predicted (TN for cat).
    

This matrix helps calculate precision, recall, and accuracy for each class.

Precision vs Recall Tradeoff with Examples

Imagine a model that detects defective products in a factory.

• High Precision: Few good products are marked defective. This means less waste but might miss some defects.
• High Recall: Most defective products are caught, but some good ones might be wrongly rejected.

Choosing between precision and recall depends on what is more costly: missing defects or wasting good products.

Good vs Bad Metric Values for Model Comparison

Good: High accuracy (e.g., 95%+), balanced precision and recall (both above 90%), and high F1 score show the model predicts well and is reliable.

Bad: High accuracy but low recall (e.g., 98% accuracy but 20% recall) means the model misses many important cases. Or very low precision means many false alarms.

Common Pitfalls in Model Comparison Metrics

• Accuracy Paradox: High accuracy can be misleading if classes are unbalanced.
• Data Leakage: Using test data in training inflates metrics falsely.
• Overfitting: Model performs well on training but poorly on new data.
• Ignoring Context: Choosing metrics without considering what matters in the real task.

Self Check: Is a Model with 98% Accuracy but 12% Recall Good?

No, this model is not good for tasks where finding positive cases is important. The 12% recall means it misses 88% of the positive cases, which can be very harmful, for example in disease detection or fraud. High accuracy here is misleading because most data might be negative cases.

Key Result

Model comparison relies on balanced metrics like precision, recall, and F1 score to choose the best model for the task.

Practice

1. What is the main reason to compare different computer vision models on the same dataset?

easy

A. To find which model performs best for the task

B. To make the code run faster

C. To use more memory

D. To increase the dataset size

Solution

1. Step 1: Understand the purpose of model comparison

   Model comparison is done to evaluate which model gives better results on the same data.

2. Step 2: Identify the goal of comparing models

   The goal is to pick the best model for the task, not to affect code speed or data size.

3. Final Answer:

   To find which model performs best for the task -> Option A

4. Quick Check:

   Model comparison = find best model [OK]

Hint: Compare models by their results on the same data [OK]

Common Mistakes:

• Thinking comparison changes dataset size
• Confusing speed with model quality
• Assuming more memory means better model

2. Which of the following code snippets correctly compares two models' accuracy on the same test data in Python?

easy

A. acc1 = model1.fit(X_test, y_test)
acc2 = model2.fit(X_test, y_test)

B. acc1 = model1.evaluate(X_test, y_test)[1]
acc2 = model2.evaluate(X_test, y_test)[1]

C. acc1 = model1.predict(X_test)
acc2 = model2.predict(X_test)

D. acc1 = model1.score(X_train)
acc2 = model2.score(X_train)

Solution

1. Step 1: Identify correct method to get accuracy

   Using evaluate on test data returns loss and accuracy; index 1 is accuracy.

2. Step 2: Check other options for correctness

   fit trains, not evaluates; predict gives predictions, not accuracy; score needs both data and labels.

3. Final Answer:

   acc1 = model1.evaluate(X_test, y_test)[1] acc2 = model2.evaluate(X_test, y_test)[1] -> Option B

4. Quick Check:

   Use evaluate() for accuracy [OK]

Hint: Use evaluate() on test data to get accuracy [OK]

Common Mistakes:

• Using fit() instead of evaluate() for accuracy
• Using predict() output as accuracy
• Evaluating on training data instead of test data

3. Given the code below, what will be printed?

acc1 = 0.85
acc2 = 0.90
if acc1 > acc2:
    print('Model 1 is better')
else:
    print('Model 2 is better')

medium

A. Model 1 is better

B. Error: comparison not possible

C. Model 2 is better

D. No output

Solution

1. Step 1: Compare accuracy values

   acc1 is 0.85 and acc2 is 0.90, so acc1 < acc2.

2. Step 2: Follow the if-else logic

   Since acc1 > acc2 is false, the else block runs printing 'Model 2 is better'.

3. Final Answer:

   Model 2 is better -> Option C

4. Quick Check:

   0.85 < 0.90 so Model 2 wins [OK]

Hint: Compare accuracy numbers directly [OK]

Common Mistakes:

• Confusing greater than with less than
• Expecting error from simple comparison
• Ignoring else block output

4. You have two models but the code below gives an error. What is the problem?

acc1 = model1.evaluate(X_test, y_test)
acc2 = model2.evaluate(X_test, y_test)
if acc1 > acc2:
    print('Model 1 better')
else:
    print('Model 2 better')

medium

A. evaluate() returns a tuple, so direct comparison fails

B. X_test and y_test are swapped

C. Missing parentheses in print statements

D. Models are not trained yet

Solution

1. Step 1: Understand evaluate() output

   evaluate() returns a tuple (loss, accuracy), not a single number.

2. Step 2: Identify why comparison fails

   Comparing tuples directly with > causes error or unexpected behavior.

3. Final Answer:

   evaluate() returns a tuple, so direct comparison fails -> Option A

4. Quick Check:

   Compare accuracy values, not tuples [OK]

Hint: Extract accuracy from evaluate() tuple before comparing [OK]

Common Mistakes:

• Comparing full evaluate() output tuples
• Swapping test data inputs
• Assuming print syntax error

5. You want to compare three models on accuracy and speed. Which approach best helps you pick the best model?

hard

A. Use the model with smallest file size regardless of accuracy

B. Pick the model with highest accuracy only, ignoring speed

C. Choose the model with fastest training time only

D. Train all models, record accuracy and inference time, then choose the best trade-off

Solution

1. Step 1: Understand multiple criteria comparison

   Comparing models on both accuracy and speed requires measuring both metrics.

2. Step 2: Choose approach balancing accuracy and speed

   Recording accuracy and inference time helps find the best trade-off for your needs.

3. Final Answer:

   Train all models, record accuracy and inference time, then choose the best trade-off -> Option D

4. Quick Check:

   Balance accuracy and speed for best model [OK]

Hint: Measure both accuracy and speed, then compare [OK]

Common Mistakes:

• Ignoring speed when accuracy matters
• Choosing fastest training but poor accuracy
• Selecting model by size alone