Evaluation metrics (accuracy, F1, confusion matrix) in NLP - Deep Dive

Overview - Evaluation metrics (accuracy, F1, confusion matrix)

What is it?

Evaluation metrics are tools to measure how well a machine learning model performs. Accuracy tells us the percentage of correct predictions. The F1 score balances how many correct positive predictions we make with how many we miss. A confusion matrix shows detailed counts of true and false predictions, helping us understand mistakes.

Why it matters

Without evaluation metrics, we wouldn't know if a model is good or bad. Imagine guessing answers on a test without checking if you got them right. Metrics help us improve models, avoid costly errors, and build trust in AI systems that affect real lives, like medical diagnosis or spam detection.

Where it fits

Before learning evaluation metrics, you should understand how models make predictions and the basics of classification. After this, you can explore advanced metrics, model tuning, and error analysis to improve model performance.

Mental Model

Core Idea

Evaluation metrics summarize a model's prediction quality by comparing its guesses to the true answers in different ways.

Think of it like...

It's like grading a test: accuracy is the overall score, the F1 score is like balancing how many questions you answered correctly and how many you skipped or got wrong, and the confusion matrix is the detailed answer sheet showing which questions you got right or wrong.

┌───────────────┐
│ Confusion     │
│ Matrix       │
│               │
│  TP | FP      │
│  FN | TN      │
└───────────────┘

Accuracy = (TP + TN) / Total
F1 = 2 * (Precision * Recall) / (Precision + Recall)
Precision = TP / (TP + FP)
Recall = TP / (TP + FN)

Build-Up - 7 Steps

1

Foundation

Understanding True and False Predictions

Concept: Introduce the basic idea of correct and incorrect predictions in classification.

When a model guesses, it can be right or wrong. If it guesses 'yes' and the true answer is 'yes', that's a True Positive (TP). If it guesses 'yes' but the true answer is 'no', that's a False Positive (FP). Similarly, False Negative (FN) is guessing 'no' when the answer is 'yes', and True Negative (TN) is guessing 'no' when the answer is 'no'.

Result

You can now label each prediction as TP, FP, FN, or TN.

Understanding these four outcomes is the foundation for all evaluation metrics in classification.

2

Foundation

Calculating Accuracy Metric

Concept: Learn how to compute accuracy from prediction outcomes.

Accuracy measures how many predictions were correct out of all predictions. Formula: Accuracy = (TP + TN) / (TP + FP + FN + TN). For example, if out of 100 predictions, 90 were correct, accuracy is 90%.

Result

You can measure overall correctness of a model's predictions.

Accuracy is simple and intuitive but can be misleading if classes are imbalanced.

3

Intermediate

Introducing Precision and Recall

🤔Before reading on: do you think precision and recall measure the same thing or different aspects of prediction quality? Commit to your answer.

Concept: Precision and recall focus on positive predictions but measure different things.

Precision tells us how many predicted positives were actually correct: Precision = TP / (TP + FP). Recall tells us how many actual positives were found: Recall = TP / (TP + FN). For example, if a model predicts 10 positives and 8 are correct, precision is 80%. If there were 20 actual positives and the model found 8, recall is 40%.

Result

You can evaluate how well a model finds positives and avoids false alarms.

Knowing precision and recall helps balance between missing positives and making false alarms, which accuracy alone can't show.

4

Intermediate

Understanding the F1 Score

🤔Before reading on: do you think the F1 score favors precision, recall, or balances both equally? Commit to your answer.

Concept: F1 score combines precision and recall into one number to balance their trade-off.

F1 score is the harmonic mean of precision and recall: F1 = 2 * (Precision * Recall) / (Precision + Recall). It is high only when both precision and recall are high. For example, if precision is 80% and recall is 40%, F1 is about 53%.

Result

You get a single metric that balances finding positives and avoiding false alarms.

F1 score is useful when you want a balance and when classes are imbalanced.

5

Intermediate

Reading the Confusion Matrix

Concept: Learn how to use the confusion matrix to see detailed prediction results.

A confusion matrix is a table showing counts of TP, FP, FN, and TN. It helps you see exactly where the model makes mistakes. For example: | | Predicted Yes | Predicted No | |----------|---------------|--------------| | Actual Yes | TP=50 | FN=10 | | Actual No | FP=5 | TN=35 | This shows the model predicted 50 true positives, missed 10 positives, made 5 false alarms, and correctly rejected 35 negatives.

Result

You can analyze model errors in detail and decide how to improve.

The confusion matrix reveals the full picture behind summary metrics like accuracy and F1.

6

Advanced

Handling Imbalanced Data with Metrics

🤔Before reading on: do you think accuracy is reliable when one class is much bigger than the other? Commit to your answer.

Concept: Explore why accuracy can be misleading when classes are imbalanced and how F1 helps.

If 95% of data is negative, a model that always predicts negative gets 95% accuracy but is useless. Precision, recall, and F1 focus on the minority class and give a better picture. For example, a model with 50% recall and 50% precision on the minority class is better than one with 95% accuracy but zero recall.

Result

You learn to choose metrics wisely based on data balance.

Understanding metric limits prevents trusting models that look good but fail on important cases.

7

Expert

Beyond Basics: Metric Trade-offs and Thresholds

🤔Before reading on: do you think changing the decision threshold affects precision and recall? Commit to your answer.

Concept: Learn how adjusting prediction thresholds changes precision, recall, and F1, and why this matters in practice.

Models often output probabilities. Choosing a cutoff (threshold) to decide positive or negative affects metrics. Raising threshold usually increases precision but lowers recall; lowering threshold does the opposite. Plotting precision-recall curves or ROC curves helps find the best balance. In production, threshold tuning aligns model behavior with business goals.

Result

You can optimize model decisions beyond fixed metrics for real needs.

Knowing how thresholds affect metrics lets you tailor models to different priorities like safety or coverage.

Under the Hood

Evaluation metrics work by comparing predicted labels to true labels for each data point. The confusion matrix counts each type of prediction outcome (TP, FP, FN, TN). Accuracy sums correct predictions over total. Precision and recall focus on positive class predictions, measuring correctness and completeness respectively. F1 score combines precision and recall using harmonic mean to balance their trade-off. Thresholds on prediction probabilities determine final labels, affecting these counts and metrics.

Why designed this way?

These metrics were designed to capture different aspects of model performance because no single number can tell the whole story. Accuracy is simple but fails with imbalanced data. Precision and recall address false positives and false negatives separately. F1 score balances these two. The confusion matrix provides a detailed breakdown to diagnose errors. This design allows flexible evaluation depending on the problem's needs.

Input Data ──> Model ──> Predictions
       │                   │
       │                   ▼
       │             Compare with
       │             True Labels
       ▼                   │
Confusion Matrix <─────────┘
       │
       ▼
Metrics Computed:
  Accuracy, Precision, Recall, F1
       │
       ▼
Decision Making and Model Improvement

Myth Busters - 4 Common Misconceptions

Quick: Does a high accuracy always mean a model is good? Commit to yes or no before reading on.

Common Belief: High accuracy means the model is performing well overall.

Reality: High accuracy can be misleading if the data is imbalanced; the model might just be guessing the majority class.

Why it matters: Relying on accuracy alone can hide poor performance on important minority classes, leading to bad decisions.

Quick: Is the F1 score simply the average of precision and recall? Commit to yes or no before reading on.

Common Belief: F1 score is the average of precision and recall values.

Reality: F1 score is the harmonic mean, not the average, which penalizes extreme differences between precision and recall.

Why it matters: Misunderstanding F1 can lead to wrong conclusions about model balance and performance.

Quick: Does the confusion matrix only apply to binary classification? Commit to yes or no before reading on.

Common Belief: Confusion matrix is only useful for two-class problems.

Reality: Confusion matrices can be extended to multi-class problems, showing counts for each class pair.

Why it matters: Ignoring multi-class confusion matrices limits understanding of complex classification tasks.

Quick: Does changing the classification threshold affect accuracy? Commit to yes or no before reading on.

Common Belief: Changing the threshold only affects precision and recall, not accuracy.

Reality: Changing the threshold changes all metrics including accuracy because it changes predicted labels.

Why it matters: Not tuning thresholds can miss opportunities to improve overall model performance.

Expert Zone

1

Precision and recall can be weighted differently depending on the cost of false positives vs false negatives in the application.

2

F1 score assumes equal importance of precision and recall; sometimes F-beta scores are used to weight one more than the other.

3

Confusion matrices can be normalized to show rates instead of counts, which helps compare models across datasets.

When NOT to use

Accuracy should not be used alone when classes are imbalanced; use precision, recall, or F1 instead. For ranking problems, use metrics like AUC-ROC. For regression tasks, use different metrics like RMSE or MAE.

Production Patterns

In real systems, evaluation metrics guide model selection and threshold tuning. Confusion matrices help diagnose specific error types. F1 score is often used in competitions and imbalanced datasets. Metrics are monitored continuously to detect model drift and trigger retraining.

Connections

Signal Detection Theory

Evaluation metrics like precision and recall correspond to concepts of hit rate and false alarm rate in signal detection.

Understanding signal detection helps grasp why precision and recall trade off and how thresholds affect decisions.

Medical Diagnostics

Metrics like sensitivity (recall) and specificity (related to true negatives) are used to evaluate medical tests, similar to precision and recall in ML.

Knowing medical test evaluation clarifies why different metrics matter depending on the cost of errors.

Quality Control in Manufacturing

Confusion matrix is like a defect detection report showing false alarms and misses in product inspection.

This connection shows how evaluation metrics help balance catching defects without wasting resources on false alarms.

Common Pitfalls

#1 Using accuracy as the only metric on imbalanced data.

Wrong approach:

accuracy = (TP + TN) / (TP + FP + FN + TN)
# Model with 95% accuracy but misses all minority class

Correct approach:

precision = TP / (TP + FP)
recall = TP / (TP + FN)
f1 = 2 * (precision * recall) / (precision + recall)
# Use these metrics to evaluate minority class performance

Root cause:

Misunderstanding that accuracy can be high even if the model ignores the minority class.

#2 Calculating F1 score as simple average of precision and recall.

Wrong approach:

f1 = (precision + recall) / 2 # Incorrect formula

Correct approach:

f1 = 2 * (precision * recall) / (precision + recall) # Correct harmonic mean

Root cause:

Confusing harmonic mean with arithmetic mean, leading to wrong F1 values.

#3 Ignoring threshold tuning and using default 0.5 cutoff.

Wrong approach:

predicted_label = 1 if probability >= 0.5 else 0
# No threshold tuning

Correct approach:

threshold = 0.3 # Adjusted based on precision-recall tradeoff
predicted_label = 1 if probability >= threshold else 0

Root cause:

Assuming default threshold is always optimal without considering application needs.

Key Takeaways

Evaluation metrics translate model predictions into meaningful numbers that tell us how well the model performs.

Accuracy is easy to understand but can be misleading when classes are imbalanced.

Precision and recall measure different aspects of positive prediction quality and must be balanced carefully.

The F1 score combines precision and recall to give a balanced single metric, especially useful for imbalanced data.

The confusion matrix provides a detailed view of prediction errors, essential for diagnosing and improving models.