Error analysis patterns in TensorFlow - Model Metrics & Evaluation

Metrics & Evaluation - Error analysis patterns

Which metric matters for error analysis patterns and WHY

Error analysis helps us find where a model makes mistakes. To do this well, we look at precision, recall, and the confusion matrix. These show us exactly what kinds of errors happen, like false alarms or missed cases. Understanding these helps us fix the model better than just looking at overall accuracy.

Confusion matrix example

      Actual \ Predicted | Positive | Negative
      -------------------|----------|---------
      Positive           |    70    |   30    
      Negative           |    10    |   90    

      Total samples = 70 + 30 + 10 + 90 = 200

      Precision = TP / (TP + FP) = 70 / (70 + 10) = 0.875
      Recall = TP / (TP + FN) = 70 / (70 + 30) = 0.7
    

This matrix shows the model predicts 70 true positives, but misses 30 positives (false negatives). It also wrongly predicts 10 negatives as positives (false positives).

Precision vs Recall tradeoff with examples

Imagine a spam email filter:

• High precision means most emails marked as spam really are spam. This avoids losing good emails.
• High recall means catching most spam emails, but might mark some good emails wrongly.

For a cancer detector:

• High recall is critical to catch all cancer cases, even if some false alarms happen.
• High precision is less important because missing cancer is worse than false alarms.

Error analysis patterns help us see which errors happen more and guide us to balance precision and recall properly.

What "good" vs "bad" metric values look like for error analysis

Good error analysis shows:

• Clear identification of error types (false positives, false negatives)
• Balanced precision and recall matching the problem needs (e.g., recall > 0.9 for cancer detection)
• Confusion matrix numbers adding up correctly and matching metrics

Bad error analysis might have:

• Ignoring false negatives or false positives
• High accuracy but very low recall or precision
• Confusion matrix numbers that don't add up or mismatch metrics

Common pitfalls in error analysis metrics

• Accuracy paradox: High accuracy can hide poor performance on rare classes.
• Data leakage: When test data leaks into training, metrics look better but model fails in real life.
• Overfitting indicators: Very high training metrics but low test metrics mean the model memorizes instead of learning.
• Ignoring class imbalance: Metrics like accuracy can be misleading if one class dominates.

Self-check question

Your model has 98% accuracy but only 12% recall on fraud cases. Is it good for production? Why or why not?

Answer: No, it is not good. The model misses 88% of fraud cases (low recall), which is dangerous because fraud is rare but important to catch. High accuracy is misleading here because most transactions are not fraud.

Key Result

Error analysis focuses on precision, recall, and confusion matrix to identify and fix specific model errors effectively.

Practice

1. What is the main purpose of error analysis in TensorFlow models?

easy

A. To find where and why the model makes mistakes

B. To increase the size of the training dataset

C. To speed up model training

D. To reduce the number of layers in the model

Solution

1. Step 1: Understand error analysis purpose

   Error analysis focuses on identifying mistakes made by the model during prediction.

2. Step 2: Compare options with purpose

   Only To find where and why the model makes mistakes correctly states that error analysis finds where and why the model makes mistakes.

3. Final Answer:

   To find where and why the model makes mistakes -> Option A

4. Quick Check:

   Error analysis = find model mistakes [OK]

Hint: Error analysis = find and understand model mistakes [OK]

Common Mistakes:

• Confusing error analysis with data augmentation
• Thinking error analysis speeds up training
• Believing error analysis changes model architecture

2. Which TensorFlow function is commonly used to compare model predictions with true labels during error analysis?

easy

A. tf.argmax()

B. tf.equal()

C. tf.reshape()

D. tf.reduce_sum()

Solution

1. Step 1: Identify function for comparing predictions and labels

   tf.equal() compares two tensors element-wise and returns True where they match.

2. Step 2: Check other options

   tf.argmax() finds index of max value, tf.reshape() changes shape, tf.reduce_sum() sums values; none compare predictions to labels.

3. Final Answer:

   tf.equal() -> Option B

4. Quick Check:

   Compare predictions and labels = tf.equal() [OK]

Hint: Use tf.equal() to check prediction correctness [OK]

Common Mistakes:

• Using tf.argmax() to compare directly
• Confusing reshape with comparison
• Summing values instead of comparing

3. Given the following TensorFlow code snippet, what will be the output?

predictions = tf.constant([1, 0, 1, 1])
labels = tf.constant([1, 1, 0, 1])
errors = tf.not_equal(predictions, labels)
print(tf.reduce_sum(tf.cast(errors, tf.int32)).numpy())

medium

A. 4

B. 1

C. 3

D. 2

Solution

1. Step 1: Compare predictions and labels element-wise

   predictions = [1,0,1,1], labels = [1,1,0,1]. Differences at index 1 (0 vs 1) and index 2 (1 vs 0).

2. Step 2: Count number of errors

   tf.not_equal returns True for mismatches. There are 2 mismatches, so sum is 2.

3. Final Answer:

   2 -> Option D

4. Quick Check:

   Count mismatches = 2 [OK]

Hint: Count mismatches with tf.not_equal and sum [OK]

Common Mistakes:

• Counting matches instead of mismatches
• Confusing tf.equal with tf.not_equal
• Forgetting to cast booleans to int before summing

4. You run this TensorFlow code to find errors but get a TypeError:

errors = tf.not_equal(predictions, labels)
error_count = tf.reduce_sum(errors).numpy()

What is the likely cause and fix?

medium

A. predictions and labels have different shapes

B. tf.reduce_sum() requires a Python list input

C. errors is boolean; cast to int before summing

D. numpy() cannot be called on TensorFlow tensors

Solution

1. Step 1: Identify error cause

   tf.not_equal returns a boolean tensor. tf.reduce_sum on booleans causes TypeError without casting.

2. Step 2: Fix by casting booleans to int

   Use tf.cast(errors, tf.int32) before tf.reduce_sum to count True values correctly.

3. Final Answer:

   errors is boolean; cast to int before summing -> Option C

4. Quick Check:

   Cast booleans to int before sum [OK]

Hint: Cast booleans to int before summing [OK]

Common Mistakes:

• Ignoring data type of errors tensor
• Assuming reduce_sum works on booleans directly
• Misunderstanding numpy() usage

5. You want to perform error analysis by grouping model errors by class label in TensorFlow. Which approach best helps identify which classes have the most errors?

hard

A. Use tf.boolean_mask to select errors per class, then count errors per class

B. Shuffle the dataset randomly before error analysis

C. Increase batch size during model training

D. Use tf.argmax on predictions without comparing to labels

Solution

1. Step 1: Understand grouping errors by class

   To find errors per class, select errors where label equals that class using tf.boolean_mask.

2. Step 2: Count errors per class

   Counting errors per class after masking helps identify classes with most mistakes.

3. Final Answer:

   Use tf.boolean_mask to select errors per class, then count errors per class -> Option A

4. Quick Check:

   Group errors by class using boolean mask [OK]

Hint: Mask errors by class label to count per class [OK]

Common Mistakes:

• Shuffling data does not group errors
• Batch size affects training, not error grouping
• Using tf.argmax alone ignores true labels