Evaluation and confusion matrix in Computer Vision - ML Experiment: Train & Evaluate

Experiment - Evaluation and confusion matrix

Problem:We have trained a simple image classifier to recognize two types of fruits: apples and oranges. The model currently shows good training accuracy but we want to understand how well it performs on new images by evaluating it with a confusion matrix.

Current Metrics:Training accuracy: 95%, Validation accuracy: 88%

Issue:We do not have detailed insight into the types of errors the model makes, such as whether it confuses apples for oranges or vice versa.

Your Task

Create and interpret a confusion matrix for the model's predictions on the validation set to better understand its errors.

Use the existing trained model and validation data.

Do not change the model architecture or training process.

Hint 1

Hint 2

Hint 3

Solution

Computer Vision

import numpy as np
import matplotlib.pyplot as plt
from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay, accuracy_score, precision_score, recall_score

# Simulated validation data and labels (0=apple, 1=orange)
y_true = np.array([0, 0, 1, 1, 0, 1, 0, 1, 1, 0])
# Simulated model predictions
y_pred = np.array([0, 1, 1, 1, 0, 0, 0, 1, 1, 0])

# Compute confusion matrix
cm = confusion_matrix(y_true, y_pred)

# Display confusion matrix
labels = ['Apple', 'Orange']
cmd = ConfusionMatrixDisplay(confusion_matrix=cm, display_labels=labels)
cmd.plot(cmap=plt.cm.Blues)
plt.title('Confusion Matrix for Fruit Classifier')
plt.show()

# Calculate metrics
accuracy = accuracy_score(y_true, y_pred) * 100
precision = precision_score(y_true, y_pred, pos_label=1) * 100
recall = recall_score(y_true, y_pred, pos_label=1) * 100

print(f'Accuracy: {accuracy:.1f}%')
print(f'Precision (Orange class): {precision:.1f}%')
print(f'Recall (Orange class): {recall:.1f}%')

Added code to compute the confusion matrix using sklearn.

Visualized the confusion matrix with labels for apples and oranges.

Calculated accuracy, precision, and recall from predictions and true labels.

Specified pos_label=1 in precision_score and recall_score to clarify the positive class.

Results Interpretation

Before: We only had overall accuracy: Training 95%, Validation 88%.

After: Confusion matrix shows 4 true apples correctly predicted, 1 apple predicted as orange, 4 true oranges correctly predicted, and 1 orange predicted as apple.

Accuracy is 80%, precision and recall for orange class are both 80%, indicating some confusion between classes.

The confusion matrix helps us see exactly where the model makes mistakes, not just the overall accuracy. This detailed view guides us to improve the model or data by focusing on specific errors.

Bonus Experiment

Try adding class weights or data augmentation to improve recall for the orange class and observe changes in the confusion matrix.

💡 Hint

Use class_weight parameter in model training or augment orange images to balance the dataset.