Evaluation metrics (accuracy, F1, confusion matrix) in NLP - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Calculate accuracy from predictions and true labels

Given the true labels and predicted labels below, what is the accuracy of the model?

NLP

true_labels = [1, 0, 1, 1, 0, 0, 1]
pred_labels = [1, 0, 0, 1, 0, 1, 1]
correct = sum(t == p for t, p in zip(true_labels, pred_labels))
accuracy = correct / len(true_labels)
print(round(accuracy, 2))

A 0.85

B 0.57

C 0.71

D 0.43

💡 Hint

Count how many predictions match the true labels, then divide by total labels.

Accuracy is the number of correct predictions divided by total predictions. Here, 5 out of 7 predictions are correct, so accuracy is 5/7 ≈ 0.71.

🧠 Conceptual

intermediate

Understanding the F1 score

Which statement best describes the F1 score in classification tasks?

A It is the average of accuracy and recall.

B It measures the ratio of correct predictions to total predictions.

C It counts the number of true negatives in the confusion matrix.

D It is the harmonic mean of precision and recall, balancing both.

💡 Hint

Think about how F1 combines two important metrics to balance false positives and false negatives.

F1 score combines precision and recall using the harmonic mean, giving a single metric that balances both false positives and false negatives.

❓ Predict Output

advanced

Confusion matrix values from predictions

Given the true and predicted labels below, what is the value of True Positives (TP) in the confusion matrix?

NLP

true_labels = [0, 1, 1, 0, 1, 0, 1, 1]
pred_labels = [0, 1, 0, 0, 1, 1, 1, 0]
TP = sum(1 for t, p in zip(true_labels, pred_labels) if t == 1 and p == 1)
print(TP)

A 3

B 2

C 4

D 1

💡 Hint

Count how many times the true label and predicted label are both 1.

True Positives are cases where the true label is 1 and the prediction is also 1. Here, that happens three times.

❓ Model Choice

advanced

Choosing metric for imbalanced data

You have a dataset where 95% of samples belong to class A and 5% to class B. Which metric is best to evaluate your model's performance on class B?

A F1 score

B Loss value

C Accuracy

D Total number of predictions

💡 Hint

Think about which metric balances false positives and false negatives well for rare classes.

Accuracy can be misleading with imbalanced data. F1 score balances precision and recall, making it better for rare classes.

❓ Metrics

expert

Calculate macro F1 score from class-wise precision and recall

Given the following precision and recall for three classes, what is the macro F1 score? Class 1: precision=0.8, recall=0.6 Class 2: precision=0.7, recall=0.7 Class 3: precision=0.9, recall=0.5

A 0.72

B 0.67

C 0.70

D 0.75

💡 Hint

Calculate F1 for each class, then average them.

F1 for each class = 2 * (precision * recall) / (precision + recall). Then average all three F1 scores for macro F1.

Practice

1. What does the accuracy metric measure in a classification model?

easy

A. The proportion of correct predictions out of all predictions

B. The balance between precision and recall

C. The number of false positives only

D. The total number of classes in the dataset

Solution

1. Step 1: Understand accuracy definition

   Accuracy is defined as the number of correct predictions divided by the total number of predictions made.

2. Step 2: Compare options with definition

   Only The proportion of correct predictions out of all predictions correctly describes accuracy as the proportion of correct predictions out of all predictions.

3. Final Answer:

   The proportion of correct predictions out of all predictions -> Option A

4. Quick Check:

   Accuracy = Correct predictions / Total predictions [OK]

Hint: Accuracy = correct predictions divided by total predictions [OK]

Common Mistakes:

• Confusing accuracy with F1 score
• Thinking accuracy measures only false positives
• Believing accuracy counts number of classes

2. Which of the following is the correct formula for F1 score?

easy

A. Precision + Recall

B. 2 * (Precision * Recall) / (Precision + Recall)

C. True Positives / Total Samples

D. True Negatives / (True Negatives + False Positives)

Solution

1. Step 1: Recall F1 score formula

   F1 score is the harmonic mean of precision and recall, calculated as 2 times their product divided by their sum.

2. Step 2: Match formula with options

   2 * (Precision * Recall) / (Precision + Recall) matches the correct formula: 2 * (Precision * Recall) / (Precision + Recall).

3. Final Answer:

   2 * (Precision * Recall) / (Precision + Recall) -> Option B

4. Quick Check:

   F1 = 2PR/(P+R) [OK]

Hint: F1 score = 2 * Precision * Recall / (Precision + Recall) [OK]

Common Mistakes:

• Adding precision and recall instead of harmonic mean
• Using true positives over total samples as F1
• Confusing F1 with specificity

3. Given the confusion matrix below for a binary classifier:

[[50, 10],
 [5, 35]]

What is the accuracy of the model?

medium

A. 75%

B. 70%

C. 90%

D. 85%

Solution

1. Step 1: Identify confusion matrix values

   True Positives (TP) = 50, False Positives (FP) = 10, False Negatives (FN) = 5, True Negatives (TN) = 35.

2. Step 2: Calculate accuracy

   Accuracy = (TP + TN) / (TP + FP + FN + TN) = (50 + 35) / (50 + 10 + 5 + 35) = 85 / 100 = 0.85 or 85%.

3. Final Answer:

   85% -> Option D

4. Quick Check:

   Accuracy = (TP+TN)/Total = 85/100 = 85% [OK]

Hint: Accuracy = (TP + TN) / total samples [OK]

Common Mistakes:

• Adding false positives or false negatives to numerator
• Calculating only TP / total samples
• Mixing up TP and TN values

4. You have this confusion matrix:

[[40, 20],
 [10, 30]]

Which line of code correctly calculates precision for the positive class?

medium

A. precision = TP / (TP + FP)

B. precision = TP / (TP + FN)

C. precision = TN / (TN + FP)

D. precision = TP / (TP + TN)

Solution

1. Step 1: Recall precision formula

   Precision is the ratio of true positives to all predicted positives: TP / (TP + FP).

2. Step 2: Match formula with options

   precision = TP / (TP + FP) correctly uses TP / (TP + FP). precision = TP / (TP + FN) uses recall formula, C and D are incorrect.

3. Final Answer:

   precision = TP / (TP + FP) -> Option A

4. Quick Check:

   Precision = TP / (TP + FP) [OK]

Hint: Precision = true positives / predicted positives [OK]

Common Mistakes:

• Using TP / (TP + FN) which is recall
• Confusing TN with TP in precision
• Dividing by TP + TN instead of TP + FP

5. A model has precision = 0.8 and recall = 0.5. What is the F1 score? Choose the closest value.

hard

A. 0.70

B. 0.65

C. 0.62

D. 0.75

Solution

1. Step 1: Recall F1 score formula

   F1 = 2 * (Precision * Recall) / (Precision + Recall) = 2 * (0.8 * 0.5) / (0.8 + 0.5).

2. Step 2: Calculate F1 score

   Calculate numerator: 2 * 0.4 = 0.8. Calculate denominator: 1.3. F1 = 0.8 / 1.3 ≈ 0.615.

3. Final Answer:

   0.62 -> Option C

4. Quick Check:

   F1 ≈ 0.62 from 0.8 precision and 0.5 recall [OK]

Hint: F1 is harmonic mean: 2PR/(P+R), plug values carefully [OK]

Common Mistakes:

• Averaging precision and recall instead of harmonic mean
• Mixing up precision and recall values
• Rounding too early causing wrong final answer