Evaluation metrics (accuracy, F1, confusion matrix) in NLP - Model Metrics & Evaluation

Metrics & Evaluation - Evaluation metrics (accuracy, F1, confusion matrix)

Which metric matters and WHY

In natural language processing (NLP), we often want to know how well our model predicts the right answers. Accuracy tells us the overall percentage of correct predictions. But accuracy alone can be misleading if the data is unbalanced.

F1 score balances two important ideas: precision (how many predicted positives are actually correct) and recall (how many actual positives the model found). This is very useful when we care about both missing important cases and avoiding false alarms.

The confusion matrix shows the counts of true positives, false positives, true negatives, and false negatives. It helps us understand exactly where the model makes mistakes.

Confusion Matrix Example

      | Predicted Positive | Predicted Negative |
      |--------------------|--------------------|
      | True Positive (TP): 50 | False Positive (FP): 5 |
      | False Negative (FN): 10 | True Negative (TN): 35 |
    

Total samples = TP + FP + TN + FN = 50 + 5 + 35 + 10 = 100

Precision vs Recall Tradeoff

Imagine a spam email detector:

• High precision means most emails marked as spam really are spam. This avoids losing good emails.
• High recall means the detector finds most spam emails, even if some good emails get caught.

Depending on what matters more, we adjust the model to favor precision or recall.

Good vs Bad Metric Values

For an NLP task like sentiment analysis:

• Good: Accuracy around 85% or higher, F1 score above 0.8, balanced precision and recall.
• Bad: Accuracy near 50% (random guessing), F1 score below 0.5, very low recall or precision indicating many missed or wrong predictions.

Common Pitfalls

• Accuracy paradox: High accuracy can hide poor performance if classes are imbalanced.
• Data leakage: When test data leaks into training, metrics look unrealistically good.
• Overfitting: Very high training accuracy but low test accuracy means the model memorizes instead of learning.

Self Check

Your NLP model has 98% accuracy but only 12% recall on the positive class (e.g., detecting spam). Is it good for production?

Answer: No, because the model misses most positive cases (spam). High accuracy is misleading here due to class imbalance. Improving recall is critical.

Key Result

F1 score balances precision and recall, providing a clearer picture than accuracy alone, especially with imbalanced NLP data.

Practice

1. What does the accuracy metric measure in a classification model?

easy

A. The proportion of correct predictions out of all predictions

B. The balance between precision and recall

C. The number of false positives only

D. The total number of classes in the dataset

Solution

1. Step 1: Understand accuracy definition

   Accuracy is defined as the number of correct predictions divided by the total number of predictions made.

2. Step 2: Compare options with definition

   Only The proportion of correct predictions out of all predictions correctly describes accuracy as the proportion of correct predictions out of all predictions.

3. Final Answer:

   The proportion of correct predictions out of all predictions -> Option A

4. Quick Check:

   Accuracy = Correct predictions / Total predictions [OK]

Hint: Accuracy = correct predictions divided by total predictions [OK]

Common Mistakes:

• Confusing accuracy with F1 score
• Thinking accuracy measures only false positives
• Believing accuracy counts number of classes

2. Which of the following is the correct formula for F1 score?

easy

A. Precision + Recall

B. 2 * (Precision * Recall) / (Precision + Recall)

C. True Positives / Total Samples

D. True Negatives / (True Negatives + False Positives)

Solution

1. Step 1: Recall F1 score formula

   F1 score is the harmonic mean of precision and recall, calculated as 2 times their product divided by their sum.

2. Step 2: Match formula with options

   2 * (Precision * Recall) / (Precision + Recall) matches the correct formula: 2 * (Precision * Recall) / (Precision + Recall).

3. Final Answer:

   2 * (Precision * Recall) / (Precision + Recall) -> Option B

4. Quick Check:

   F1 = 2PR/(P+R) [OK]

Hint: F1 score = 2 * Precision * Recall / (Precision + Recall) [OK]

Common Mistakes:

• Adding precision and recall instead of harmonic mean
• Using true positives over total samples as F1
• Confusing F1 with specificity

3. Given the confusion matrix below for a binary classifier:

[[50, 10],
 [5, 35]]

What is the accuracy of the model?

medium

A. 75%

B. 70%

C. 90%

D. 85%

Solution

1. Step 1: Identify confusion matrix values

   True Positives (TP) = 50, False Positives (FP) = 10, False Negatives (FN) = 5, True Negatives (TN) = 35.

2. Step 2: Calculate accuracy

   Accuracy = (TP + TN) / (TP + FP + FN + TN) = (50 + 35) / (50 + 10 + 5 + 35) = 85 / 100 = 0.85 or 85%.

3. Final Answer:

   85% -> Option D

4. Quick Check:

   Accuracy = (TP+TN)/Total = 85/100 = 85% [OK]

Hint: Accuracy = (TP + TN) / total samples [OK]

Common Mistakes:

• Adding false positives or false negatives to numerator
• Calculating only TP / total samples
• Mixing up TP and TN values

4. You have this confusion matrix:

[[40, 20],
 [10, 30]]

Which line of code correctly calculates precision for the positive class?

medium

A. precision = TP / (TP + FP)

B. precision = TP / (TP + FN)

C. precision = TN / (TN + FP)

D. precision = TP / (TP + TN)

Solution

1. Step 1: Recall precision formula

   Precision is the ratio of true positives to all predicted positives: TP / (TP + FP).

2. Step 2: Match formula with options

   precision = TP / (TP + FP) correctly uses TP / (TP + FP). precision = TP / (TP + FN) uses recall formula, C and D are incorrect.

3. Final Answer:

   precision = TP / (TP + FP) -> Option A

4. Quick Check:

   Precision = TP / (TP + FP) [OK]

Hint: Precision = true positives / predicted positives [OK]

Common Mistakes:

• Using TP / (TP + FN) which is recall
• Confusing TN with TP in precision
• Dividing by TP + TN instead of TP + FP

5. A model has precision = 0.8 and recall = 0.5. What is the F1 score? Choose the closest value.

hard

A. 0.70

B. 0.65

C. 0.62

D. 0.75

Solution

1. Step 1: Recall F1 score formula

   F1 = 2 * (Precision * Recall) / (Precision + Recall) = 2 * (0.8 * 0.5) / (0.8 + 0.5).

2. Step 2: Calculate F1 score

   Calculate numerator: 2 * 0.4 = 0.8. Calculate denominator: 1.3. F1 = 0.8 / 1.3 ≈ 0.615.

3. Final Answer:

   0.62 -> Option C

4. Quick Check:

   F1 ≈ 0.62 from 0.8 precision and 0.5 recall [OK]

Hint: F1 is harmonic mean: 2PR/(P+R), plug values carefully [OK]

Common Mistakes:

• Averaging precision and recall instead of harmonic mean
• Mixing up precision and recall values
• Rounding too early causing wrong final answer