Error analysis patterns in TensorFlow - ML Experiment: Train & Evaluate

Experiment - Error analysis patterns

Problem:You trained a neural network to classify images into 3 categories. The training accuracy is 95%, but validation accuracy is only 70%. You want to understand why the model makes mistakes.

Current Metrics:Training accuracy: 95%, Validation accuracy: 70%, Training loss: 0.15, Validation loss: 0.85

Issue:The model overfits the training data and performs poorly on validation data. You need to analyze the errors to find patterns causing low validation accuracy.

Your Task

Perform error analysis on the validation set predictions to identify common patterns in misclassified images and suggest improvements.

Use TensorFlow and Python only.

Do not change the model architecture or training hyperparameters yet.

Focus on analyzing errors and visualizing them.

Hint 1

Hint 2

Hint 3

Hint 4

Solution

TensorFlow

import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay

# Assume model and validation data are loaded
# model: trained TensorFlow model
# val_images, val_labels: validation dataset

# Get predictions
val_pred_probs = model.predict(val_images)
val_preds = np.argmax(val_pred_probs, axis=1)
true_labels = np.argmax(val_labels, axis=1)

# Calculate confusion matrix
cm = confusion_matrix(true_labels, val_preds)

# Display confusion matrix
disp = ConfusionMatrixDisplay(confusion_matrix=cm, display_labels=["Class 0", "Class 1", "Class 2"])
disp.plot(cmap=plt.cm.Blues)
plt.title("Confusion Matrix on Validation Set")
plt.show()

# Calculate per-class accuracy
per_class_acc = cm.diagonal() / cm.sum(axis=1)
for i, acc in enumerate(per_class_acc):
    print(f"Accuracy for Class {i}: {acc*100:.2f}%")

# Find indices of misclassified samples
misclassified_indices = np.where(val_preds != true_labels)[0]

# Visualize some misclassified images
num_to_show = 6
plt.figure(figsize=(12, 6))
for i, idx in enumerate(misclassified_indices[:num_to_show]):
    plt.subplot(2, 3, i+1)
    plt.imshow(val_images[idx])
    plt.title(f"True: {true_labels[idx]}, Pred: {val_preds[idx]}")
    plt.axis('off')
plt.suptitle("Examples of Misclassified Validation Images")
plt.show()

Added code to predict on validation data and compare with true labels.

Computed and visualized confusion matrix to identify which classes are confused.

Calculated per-class accuracy to find weak classes.

Displayed sample misclassified images to visually inspect error patterns.

Results Interpretation

Before error analysis: You only knew overall validation accuracy was low (70%).

After error analysis: You see which classes the model confuses most from the confusion matrix. Per-class accuracy shows which classes need improvement. Visualizing misclassified images helps identify if errors are due to ambiguous images, poor lighting, or similar features.

Error analysis helps you understand why your model makes mistakes. Instead of guessing, you use data-driven insights to guide improvements like collecting more data for weak classes, augmenting images, or adjusting the model.

Bonus Experiment

Try adding data augmentation to the training data and retrain the model to see if validation accuracy improves for the weak classes identified.

💡 Hint

Use TensorFlow's ImageDataGenerator or tf.image functions to add random flips, rotations, or brightness changes during training.

Practice

1. What is the main purpose of error analysis in TensorFlow models?

easy

A. To find where and why the model makes mistakes

B. To increase the size of the training dataset

C. To speed up model training

D. To reduce the number of layers in the model

Solution

1. Step 1: Understand error analysis purpose

   Error analysis focuses on identifying mistakes made by the model during prediction.

2. Step 2: Compare options with purpose

   Only To find where and why the model makes mistakes correctly states that error analysis finds where and why the model makes mistakes.

3. Final Answer:

   To find where and why the model makes mistakes -> Option A

4. Quick Check:

   Error analysis = find model mistakes [OK]

Hint: Error analysis = find and understand model mistakes [OK]

Common Mistakes:

• Confusing error analysis with data augmentation
• Thinking error analysis speeds up training
• Believing error analysis changes model architecture

2. Which TensorFlow function is commonly used to compare model predictions with true labels during error analysis?

easy

A. tf.argmax()

B. tf.equal()

C. tf.reshape()

D. tf.reduce_sum()

Solution

1. Step 1: Identify function for comparing predictions and labels

   tf.equal() compares two tensors element-wise and returns True where they match.

2. Step 2: Check other options

   tf.argmax() finds index of max value, tf.reshape() changes shape, tf.reduce_sum() sums values; none compare predictions to labels.

3. Final Answer:

   tf.equal() -> Option B

4. Quick Check:

   Compare predictions and labels = tf.equal() [OK]

Hint: Use tf.equal() to check prediction correctness [OK]

Common Mistakes:

• Using tf.argmax() to compare directly
• Confusing reshape with comparison
• Summing values instead of comparing

3. Given the following TensorFlow code snippet, what will be the output?

predictions = tf.constant([1, 0, 1, 1])
labels = tf.constant([1, 1, 0, 1])
errors = tf.not_equal(predictions, labels)
print(tf.reduce_sum(tf.cast(errors, tf.int32)).numpy())

medium

A. 4

B. 1

C. 3

D. 2

Solution

1. Step 1: Compare predictions and labels element-wise

   predictions = [1,0,1,1], labels = [1,1,0,1]. Differences at index 1 (0 vs 1) and index 2 (1 vs 0).

2. Step 2: Count number of errors

   tf.not_equal returns True for mismatches. There are 2 mismatches, so sum is 2.

3. Final Answer:

   2 -> Option D

4. Quick Check:

   Count mismatches = 2 [OK]

Hint: Count mismatches with tf.not_equal and sum [OK]

Common Mistakes:

• Counting matches instead of mismatches
• Confusing tf.equal with tf.not_equal
• Forgetting to cast booleans to int before summing

4. You run this TensorFlow code to find errors but get a TypeError:

errors = tf.not_equal(predictions, labels)
error_count = tf.reduce_sum(errors).numpy()

What is the likely cause and fix?

medium

A. predictions and labels have different shapes

B. tf.reduce_sum() requires a Python list input

C. errors is boolean; cast to int before summing

D. numpy() cannot be called on TensorFlow tensors

Solution

1. Step 1: Identify error cause

   tf.not_equal returns a boolean tensor. tf.reduce_sum on booleans causes TypeError without casting.

2. Step 2: Fix by casting booleans to int

   Use tf.cast(errors, tf.int32) before tf.reduce_sum to count True values correctly.

3. Final Answer:

   errors is boolean; cast to int before summing -> Option C

4. Quick Check:

   Cast booleans to int before sum [OK]

Hint: Cast booleans to int before summing [OK]

Common Mistakes:

• Ignoring data type of errors tensor
• Assuming reduce_sum works on booleans directly
• Misunderstanding numpy() usage

5. You want to perform error analysis by grouping model errors by class label in TensorFlow. Which approach best helps identify which classes have the most errors?

hard

A. Use tf.boolean_mask to select errors per class, then count errors per class

B. Shuffle the dataset randomly before error analysis

C. Increase batch size during model training

D. Use tf.argmax on predictions without comparing to labels

Solution

1. Step 1: Understand grouping errors by class

   To find errors per class, select errors where label equals that class using tf.boolean_mask.

2. Step 2: Count errors per class

   Counting errors per class after masking helps identify classes with most mistakes.

3. Final Answer:

   Use tf.boolean_mask to select errors per class, then count errors per class -> Option A

4. Quick Check:

   Group errors by class using boolean mask [OK]

Hint: Mask errors by class label to count per class [OK]

Common Mistakes:

• Shuffling data does not group errors
• Batch size affects training, not error grouping
• Using tf.argmax alone ignores true labels