Evaluation and confusion matrix in Computer Vision - ML Experiment: Train & Evaluate

Experiment - Evaluation and confusion matrix

Problem:We have trained a simple image classifier to recognize two types of fruits: apples and oranges. The model currently shows good training accuracy but we want to understand how well it performs on new images by evaluating it with a confusion matrix.

Current Metrics:Training accuracy: 95%, Validation accuracy: 88%

Issue:We do not have detailed insight into the types of errors the model makes, such as whether it confuses apples for oranges or vice versa.

Your Task

Create and interpret a confusion matrix for the model's predictions on the validation set to better understand its errors.

Use the existing trained model and validation data.

Do not change the model architecture or training process.

Hint 1

Hint 2

Hint 3

Solution

Computer Vision

import numpy as np
import matplotlib.pyplot as plt
from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay, accuracy_score, precision_score, recall_score

# Simulated validation data and labels (0=apple, 1=orange)
y_true = np.array([0, 0, 1, 1, 0, 1, 0, 1, 1, 0])
# Simulated model predictions
y_pred = np.array([0, 1, 1, 1, 0, 0, 0, 1, 1, 0])

# Compute confusion matrix
cm = confusion_matrix(y_true, y_pred)

# Display confusion matrix
labels = ['Apple', 'Orange']
cmd = ConfusionMatrixDisplay(confusion_matrix=cm, display_labels=labels)
cmd.plot(cmap=plt.cm.Blues)
plt.title('Confusion Matrix for Fruit Classifier')
plt.show()

# Calculate metrics
accuracy = accuracy_score(y_true, y_pred) * 100
precision = precision_score(y_true, y_pred, pos_label=1) * 100
recall = recall_score(y_true, y_pred, pos_label=1) * 100

print(f'Accuracy: {accuracy:.1f}%')
print(f'Precision (Orange class): {precision:.1f}%')
print(f'Recall (Orange class): {recall:.1f}%')

Added code to compute the confusion matrix using sklearn.

Visualized the confusion matrix with labels for apples and oranges.

Calculated accuracy, precision, and recall from predictions and true labels.

Specified pos_label=1 in precision_score and recall_score to clarify the positive class.

Results Interpretation

Before: We only had overall accuracy: Training 95%, Validation 88%.

After: Confusion matrix shows 4 true apples correctly predicted, 1 apple predicted as orange, 4 true oranges correctly predicted, and 1 orange predicted as apple.

Accuracy is 80%, precision and recall for orange class are both 80%, indicating some confusion between classes.

The confusion matrix helps us see exactly where the model makes mistakes, not just the overall accuracy. This detailed view guides us to improve the model or data by focusing on specific errors.

Bonus Experiment

Try adding class weights or data augmentation to improve recall for the orange class and observe changes in the confusion matrix.

💡 Hint

Use class_weight parameter in model training or augment orange images to balance the dataset.

Practice

1. What does a confusion matrix help you understand in a classification model?

easy

A. The speed of the model during training

B. How well the model predicts each class by showing true and false predictions

C. The number of layers in the model

D. The size of the input images

Solution

1. Step 1: Understand the purpose of a confusion matrix

   A confusion matrix shows counts of correct and incorrect predictions for each class, helping evaluate classification performance.

2. Step 2: Match the description to the options

   Only How well the model predicts each class by showing true and false predictions describes this purpose correctly, while others relate to unrelated model aspects.

3. Final Answer:

   How well the model predicts each class by showing true and false predictions -> Option B

4. Quick Check:

   Confusion matrix = True/False predictions summary [OK]

Hint: Confusion matrix shows correct vs wrong class predictions [OK]

Common Mistakes:

• Confusing confusion matrix with model speed
• Thinking it shows model architecture details
• Assuming it shows input data size

2. Which of the following is the correct way to create a confusion matrix using scikit-learn in Python?

easy

A. confusion_matrix(y_pred)

B. confusionMatrix(y_true, y_pred)

C. conf_matrix(y_pred, y_true)

D. confusion_matrix(y_true, y_pred)

Solution

1. Step 1: Recall the scikit-learn function signature

   The function to create a confusion matrix is confusion_matrix(y_true, y_pred) with true labels first, then predicted labels.

2. Step 2: Check each option for correctness

   confusion_matrix(y_true, y_pred) matches the correct function and argument order. Options B, C, and D have wrong names or argument orders.

3. Final Answer:

   confusion_matrix(y_true, y_pred) -> Option D

4. Quick Check:

   Correct function name and argument order [OK]

Hint: Use exact function name and order: confusion_matrix(true, pred) [OK]

Common Mistakes:

• Using wrong function name capitalization
• Swapping true and predicted labels
• Passing only one argument

3. Given the following code, what will be the output confusion matrix?

from sklearn.metrics import confusion_matrix

y_true = [0, 1, 0, 1, 0, 1, 1]
y_pred = [0, 0, 0, 1, 0, 1, 1]

cm = confusion_matrix(y_true, y_pred)
print(cm)

medium

A. [[3 0]
[1 3]]

B. [[2 1]
[0 4]]

C. [[3 1]
[0 3]]

D. [[4 0]
[1 2]]

Solution

1. Step 1: Count true positives and negatives

   Class 0 true positives: y_true=0 and y_pred=0 occur 3 times; false negatives: y_true=1 but y_pred=0 occur once.

2. Step 2: Build confusion matrix

   Matrix rows = true labels, columns = predicted labels. So cm = [[3,0],[1,3]] matches counts.

3. Final Answer:

   [[3 0] [1 3]] -> Option A

4. Quick Check:

   Count matches matrix entries [OK]

Hint: Count true/pred pairs carefully to fill matrix [OK]

Common Mistakes:

• Mixing rows and columns order
• Counting predicted labels as true labels
• Ignoring zero counts

4. You wrote this code but got an error:

from sklearn.metrics import confusion_matrix

cm = confusion_matrix(y_pred, y_true)
print(cm)

What is the likely cause of the error?

medium

A. Using print instead of return

B. Missing import statement for confusion_matrix

C. Swapped y_pred and y_true arguments causing shape mismatch

D. y_pred and y_true are not defined variables

Solution

1. Step 1: Check argument order for confusion_matrix

   The function expects y_true first, then y_pred. Swapping them can cause errors or wrong results.

2. Step 2: Analyze the error cause

   Since import is present and print is valid, the likely cause is swapped arguments causing shape or value errors.

3. Final Answer:

   Swapped y_pred and y_true arguments causing shape mismatch -> Option C

4. Quick Check:

   Correct argument order is true labels first [OK]

Hint: Always pass true labels first, predicted second [OK]

Common Mistakes:

• Swapping true and predicted labels
• Forgetting to import confusion_matrix
• Using undefined variables

5. You have a 3-class image classifier with classes A, B, and C. The confusion matrix is:

[[5 2 0]
 [1 7 1]
 [0 2 6]]

What is the precision for class B?

hard

A. 7 / (2 + 7 + 2) = 0.58

B. 7 / (1 + 7 + 1) = 0.7

C. 7 / (5 + 1 + 0) = 0.7

D. 7 / (7 + 1 + 2) = 0.58

Solution

1. Step 1: Identify precision formula for class B

   Precision = True Positives for B / (All predicted as B). True Positives = cm[1][1] = 7.

2. Step 2: Calculate total predicted as B

   Sum column 1: cm[0][1]=2 + cm[1][1]=7 + cm[2][1]=2 = 11. So precision = 7/11 ≈ 0.636, closest to 0.58 in 7 / (2 + 7 + 2) = 0.58.

3. Final Answer:

   7 / (2 + 7 + 2) = 0.58 -> Option A

4. Quick Check:

   Precision = TP / predicted positives [OK]

Hint: Precision = TP / sum of predicted class column [OK]

Common Mistakes:

• Using row sums instead of column sums
• Confusing precision with recall
• Ignoring off-diagonal values in predicted class column