Evaluation metrics (accuracy, F1, confusion matrix) in NLP - Model Pipeline Trace

Model Pipeline - Evaluation metrics (accuracy, F1, confusion matrix)

This pipeline shows how a text classification model is evaluated using accuracy, F1 score, and confusion matrix. These metrics help us understand how well the model predicts categories.

Data Flow - 4 Stages

1Raw Text Data

1000 rows x 1 column→Collect sentences with labels (e.g., positive/negative)→1000 rows x 2 columns

Sentence: 'I love this movie', Label: positive

↓

2Text Preprocessing

1000 rows x 2 columns→Clean text, tokenize, convert to vectors→1000 rows x 100 features

Vector for 'I love this movie' with 100 numbers

↓

3Model Prediction

1000 rows x 100 features→Model predicts labels for each input→1000 rows x 1 column

Predicted label: positive

↓

4Evaluation Metrics Calculation

1000 rows x 2 columns (true and predicted labels)→Calculate accuracy, F1 score, confusion matrix→Summary metrics and matrix

Accuracy: 0.92, F1: 0.93, Confusion matrix: [[420, 30], [50, 500]]

Training Trace - Epoch by Epoch

Loss
0.7 | *
0.6 | **
0.5 | ***
0.4 | ****
0.3 | *****
    +------------
     1 2 3 4 5 Epochs

Epoch	Loss ↓	Accuracy ↑	Observation
1	0.65	0.6	Model starts learning, accuracy is low
2	0.5	0.72	Loss decreases, accuracy improves
3	0.4	0.78	Model is learning patterns better
4	0.32	0.83	Good improvement in accuracy
5	0.28	0.85	Model converges with stable accuracy

Prediction Trace - 5 Layers

Layer 1: Input Text

Layer 2: Text Vectorization

Layer 3: Model Prediction

Layer 4: Class Decision

Layer 5: Evaluation Metrics

Model Quiz - 3 Questions

What does accuracy measure in this model?

A The number of features used in the model

B The average length of input sentences

C The percentage of correct predictions out of all predictions

D The time taken to train the model

Key Insight

Accuracy alone can be misleading if classes are imbalanced. Using F1 score and confusion matrix together gives a clearer picture of model performance, especially in text classification.

Practice

1. What does the accuracy metric measure in a classification model?

easy

A. The proportion of correct predictions out of all predictions

B. The balance between precision and recall

C. The number of false positives only

D. The total number of classes in the dataset

Solution

1. Step 1: Understand accuracy definition

   Accuracy is defined as the number of correct predictions divided by the total number of predictions made.

2. Step 2: Compare options with definition

   Only The proportion of correct predictions out of all predictions correctly describes accuracy as the proportion of correct predictions out of all predictions.

3. Final Answer:

   The proportion of correct predictions out of all predictions -> Option A

4. Quick Check:

   Accuracy = Correct predictions / Total predictions [OK]

Hint: Accuracy = correct predictions divided by total predictions [OK]

Common Mistakes:

• Confusing accuracy with F1 score
• Thinking accuracy measures only false positives
• Believing accuracy counts number of classes

2. Which of the following is the correct formula for F1 score?

easy

A. Precision + Recall

B. 2 * (Precision * Recall) / (Precision + Recall)

C. True Positives / Total Samples

D. True Negatives / (True Negatives + False Positives)

Solution

1. Step 1: Recall F1 score formula

   F1 score is the harmonic mean of precision and recall, calculated as 2 times their product divided by their sum.

2. Step 2: Match formula with options

   2 * (Precision * Recall) / (Precision + Recall) matches the correct formula: 2 * (Precision * Recall) / (Precision + Recall).

3. Final Answer:

   2 * (Precision * Recall) / (Precision + Recall) -> Option B

4. Quick Check:

   F1 = 2PR/(P+R) [OK]

Hint: F1 score = 2 * Precision * Recall / (Precision + Recall) [OK]

Common Mistakes:

• Adding precision and recall instead of harmonic mean
• Using true positives over total samples as F1
• Confusing F1 with specificity

3. Given the confusion matrix below for a binary classifier:

[[50, 10],
 [5, 35]]

What is the accuracy of the model?

medium

A. 75%

B. 70%

C. 90%

D. 85%

Solution

1. Step 1: Identify confusion matrix values

   True Positives (TP) = 50, False Positives (FP) = 10, False Negatives (FN) = 5, True Negatives (TN) = 35.

2. Step 2: Calculate accuracy

   Accuracy = (TP + TN) / (TP + FP + FN + TN) = (50 + 35) / (50 + 10 + 5 + 35) = 85 / 100 = 0.85 or 85%.

3. Final Answer:

   85% -> Option D

4. Quick Check:

   Accuracy = (TP+TN)/Total = 85/100 = 85% [OK]

Hint: Accuracy = (TP + TN) / total samples [OK]

Common Mistakes:

• Adding false positives or false negatives to numerator
• Calculating only TP / total samples
• Mixing up TP and TN values

4. You have this confusion matrix:

[[40, 20],
 [10, 30]]

Which line of code correctly calculates precision for the positive class?

medium

A. precision = TP / (TP + FP)

B. precision = TP / (TP + FN)

C. precision = TN / (TN + FP)

D. precision = TP / (TP + TN)

Solution

1. Step 1: Recall precision formula

   Precision is the ratio of true positives to all predicted positives: TP / (TP + FP).

2. Step 2: Match formula with options

   precision = TP / (TP + FP) correctly uses TP / (TP + FP). precision = TP / (TP + FN) uses recall formula, C and D are incorrect.

3. Final Answer:

   precision = TP / (TP + FP) -> Option A

4. Quick Check:

   Precision = TP / (TP + FP) [OK]

Hint: Precision = true positives / predicted positives [OK]

Common Mistakes:

• Using TP / (TP + FN) which is recall
• Confusing TN with TP in precision
• Dividing by TP + TN instead of TP + FP

5. A model has precision = 0.8 and recall = 0.5. What is the F1 score? Choose the closest value.

hard

A. 0.70

B. 0.65

C. 0.62

D. 0.75

Solution

1. Step 1: Recall F1 score formula

   F1 = 2 * (Precision * Recall) / (Precision + Recall) = 2 * (0.8 * 0.5) / (0.8 + 0.5).

2. Step 2: Calculate F1 score

   Calculate numerator: 2 * 0.4 = 0.8. Calculate denominator: 1.3. F1 = 0.8 / 1.3 ≈ 0.615.

3. Final Answer:

   0.62 -> Option C

4. Quick Check:

   F1 ≈ 0.62 from 0.8 precision and 0.5 recall [OK]

Hint: F1 is harmonic mean: 2PR/(P+R), plug values carefully [OK]

Common Mistakes:

• Averaging precision and recall instead of harmonic mean
• Mixing up precision and recall values
• Rounding too early causing wrong final answer