Model evaluation best practices in Computer Vision - Cheat Sheet & Quick Revision

Recall & Review

beginner

Why is it important to split your dataset into training, validation, and test sets?

Splitting the dataset helps to train the model on one part (training), tune parameters on another (validation), and finally evaluate performance on unseen data (test) to avoid overfitting and get a realistic measure of how the model will perform in real life.

beginner

What does 'overfitting' mean in model evaluation?

Overfitting happens when a model learns the training data too well, including noise and details that don't generalize. This causes poor performance on new, unseen data.

intermediate

What is the purpose of using metrics like accuracy, precision, recall, and F1-score in computer vision?

These metrics help measure how well the model predicts. Accuracy shows overall correctness, precision measures how many predicted positives are true, recall shows how many actual positives were found, and F1-score balances precision and recall.

intermediate

Why should you use cross-validation in model evaluation?

Cross-validation splits data into multiple parts and trains/tests the model several times. This gives a better estimate of model performance by reducing bias from a single train-test split.

beginner

What is the difference between validation data and test data?

Validation data is used during model training to tune parameters and make decisions. Test data is kept separate and used only once at the end to evaluate the final model's performance.

What is the main goal of splitting data into training and test sets?

A To check how well the model performs on unseen data

B To make the training faster

C To increase the size of the dataset

D To reduce the number of features

Splitting data allows us to test the model on data it hasn't seen before, which shows how well it will perform in real situations.

Which metric balances precision and recall in classification tasks?

A F1-score

B Accuracy

C Loss

D Mean Squared Error

F1-score combines precision and recall into a single metric to balance false positives and false negatives.

What does overfitting cause in a model?

A Better performance on new data

B Poor performance on new, unseen data

C Poor performance on training data

D Faster training

Overfitting means the model fits training data too closely and fails to generalize to new data.

Why is cross-validation useful?

A It increases dataset size

B It removes irrelevant features

C It speeds up training

D It reduces bias in performance estimates

Cross-validation tests the model multiple times on different data splits to give a more reliable performance estimate.

When should you use the test dataset?

A To increase training speed

B During model training to adjust parameters

C To evaluate the final model after training

D To create new features

Test data is used only once after training to check how well the model performs on unseen data.

Explain why splitting data into training, validation, and test sets is important in model evaluation.

Think about how each set helps the model learn and be tested fairly.

Describe the difference between precision and recall and why both are important in evaluating a computer vision model.

Consider how mistakes in predictions affect model usefulness.

Practice

1. Why is it important to use a separate test set when evaluating a computer vision model?

easy

A. To check how well the model performs on new, unseen data

B. To make the training process faster

C. To increase the size of the training data

D. To reduce the number of model parameters

Solution

1. Step 1: Understand the purpose of a test set

   The test set is data the model has never seen before, used to check real-world performance.

2. Step 2: Compare test set role with other options

   Options B, C, and D do not relate to evaluation but to training or model design.

3. Final Answer:

   To check how well the model performs on new, unseen data -> Option A

4. Quick Check:

   Test set = unseen data check [OK]

Hint: Test set = new data to check model accuracy [OK]

Common Mistakes:

• Confusing test set with training set
• Thinking test set speeds up training
• Believing test set changes model size

2. Which of the following is the correct way to split data for model evaluation in Python using scikit-learn?

easy

A. split_train_test(data, 0.2)

B. train_test_split(data, test_size=0.2, random_state=42)

C. train_test(data, 0.2)

D. test_train_split(data, 0.2)

Solution

1. Step 1: Recall the correct function name in scikit-learn

   The function to split data is called train_test_split with parameters like test_size and random_state.

2. Step 2: Check the options for correct syntax

   Only train_test_split(data, test_size=0.2, random_state=42) uses the correct function name and parameters; others are invalid or do not exist.

3. Final Answer:

   train_test_split(data, test_size=0.2, random_state=42) -> Option B

4. Quick Check:

   Correct function = train_test_split [OK]

Hint: Remember scikit-learn function: train_test_split [OK]

Common Mistakes:

• Using wrong function names
• Missing required parameters
• Confusing order of train and test

3. Given the following code snippet, what will be the printed accuracy?

from sklearn.metrics import accuracy_score
true_labels = [1, 0, 1, 1, 0]
pred_labels = [1, 0, 0, 1, 0]
accuracy = accuracy_score(true_labels, pred_labels)
print("{:.2f}".format(round(accuracy, 2)))

medium

A. 0.60

B. 0.40

C. 0.80

D. 1.00

Solution

1. Step 1: Compare true and predicted labels

   True: [1, 0, 1, 1, 0], Predicted: [1, 0, 0, 1, 0]. Matches at positions 0,1,3,4 (4 correct out of 5).

2. Step 2: Calculate accuracy

   Accuracy = correct predictions / total = 4/5 = 0.8. Rounded to 2 decimals is 0.80.

3. Final Answer:

   0.80 -> Option C

4. Quick Check:

   Accuracy = 4/5 = 0.80 [OK]

Hint: Count matches, divide by total labels [OK]

Common Mistakes:

• Counting wrong matches
• Not rounding accuracy
• Confusing accuracy with precision

4. You trained a model but the test accuracy is much higher than the training accuracy. What is the most likely issue?

medium

A. Data leakage between training and test sets

B. Model is underfitting the training data

C. Test set is too small

D. Training data is too large

Solution

1. Step 1: Understand unusual accuracy pattern

   Test accuracy higher than training is unusual and often means test data was seen during training.

2. Step 2: Identify cause from options

   Data leakage means test data accidentally used in training, causing inflated test accuracy.

3. Final Answer:

   Data leakage between training and test sets -> Option A

4. Quick Check:

   High test accuracy > training = data leakage [OK]

Hint: High test accuracy than train? Check data leakage [OK]

Common Mistakes:

• Assuming underfitting causes higher test accuracy
• Ignoring data leakage possibility
• Blaming test set size without evidence

5. You want to evaluate a computer vision model for detecting rare objects in images. Which evaluation metric is best to use and why?

hard

A. Confusion matrix, because it shows training time

B. Accuracy, because it shows overall correct predictions

C. Mean Squared Error, because it measures prediction error

D. F1 score, because it balances precision and recall for imbalanced data

Solution

1. Step 1: Understand the problem of rare object detection

   Rare objects mean data is imbalanced; many negatives, few positives.

2. Step 2: Choose metric suitable for imbalanced data

   F1 score balances precision (correct positive predictions) and recall (finding all positives), ideal for rare classes.

3. Final Answer:

   F1 score, because it balances precision and recall for imbalanced data -> Option D

4. Quick Check:

   Rare class? Use F1 score [OK]

Hint: Rare classes? Use F1 score for balance [OK]

Common Mistakes:

• Using accuracy which hides imbalance
• Confusing regression metrics with classification
• Misunderstanding confusion matrix purpose