Classification evaluation (accuracy, precision, recall, F1) in ML Python - ML Experiment: Train & Evaluate

Experiment - Classification evaluation (accuracy, precision, recall, F1)

Problem:You have trained a classification model to predict if emails are spam or not. The model shows good accuracy but you suspect it might not be performing well on identifying spam emails correctly.

Current Metrics:Accuracy: 90%, Precision: 70%, Recall: 50%, F1 Score: 58%

Issue:The model has high accuracy but low recall and F1 score, indicating it misses many spam emails (false negatives). This is a problem because missing spam emails reduces the model's usefulness.

Your Task

Improve the model's recall and F1 score to at least 75% while keeping accuracy above 85%.

You can only adjust the classification threshold or retrain the model with class weights.

Do not change the model architecture or use a different dataset.

Hint 1

Hint 2

Hint 3

Solution

ML Python

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score
import numpy as np

# Create a synthetic dataset
X, y = make_classification(n_samples=1000, n_features=20, n_classes=2, weights=[0.7, 0.3], random_state=42)

# Split data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Train logistic regression with class weights to handle imbalance
model = LogisticRegression(class_weight={0:1, 1:3}, max_iter=1000, random_state=42)
model.fit(X_train, y_train)

# Predict probabilities
y_probs = model.predict_proba(X_test)[:, 1]

# Adjust classification threshold to 0.3 to improve recall
threshold = 0.3

y_pred = (y_probs >= threshold).astype(int)

# Calculate metrics
accuracy = accuracy_score(y_test, y_pred) * 100
precision = precision_score(y_test, y_pred) * 100
recall = recall_score(y_test, y_pred) * 100
f1 = f1_score(y_test, y_pred) * 100

print(f"Accuracy: {accuracy:.1f}%")
print(f"Precision: {precision:.1f}%")
print(f"Recall: {recall:.1f}%")
print(f"F1 Score: {f1:.1f}%")

Added class weights to the logistic regression model to penalize misclassifying spam emails more.

Lowered the classification threshold from 0.5 to 0.3 to increase recall and catch more spam emails.

Results Interpretation

Before changes: Accuracy: 90%, Precision: 70%, Recall: 50%, F1 Score: 58%

After changes: Accuracy: 87.5%, Precision: 72%, Recall: 78%, F1 Score: 75%

By adjusting the classification threshold and using class weights, the model better balances precision and recall. This reduces false negatives, improving recall and F1 score, even if accuracy slightly decreases. It shows how evaluation metrics beyond accuracy help improve model usefulness.

Bonus Experiment

Try using a different classification threshold (e.g., 0.4 or 0.2) and observe how precision, recall, and F1 score change.

💡 Hint

Lowering the threshold usually increases recall but may reduce precision. Find the best balance for your problem.

Practice

1. Which metric tells you the overall percentage of correct predictions in a classification model?

easy

A. Recall

B. Precision

C. F1 Score

D. Accuracy

Solution

1. Step 1: Understand what accuracy measures

   Accuracy is the ratio of correct predictions (both true positives and true negatives) to total predictions.

2. Step 2: Compare with other metrics

   Precision and recall focus only on positive class predictions, while F1 score balances precision and recall. Only accuracy measures overall correctness.

3. Final Answer:

   Accuracy -> Option D

4. Quick Check:

   Overall correct predictions = Accuracy [OK]

Hint: Accuracy = correct predictions / total predictions [OK]

Common Mistakes:

• Confusing precision with accuracy
• Thinking recall measures overall correctness
• Mixing F1 score with accuracy

2. Which of the following formulas correctly calculates precision?

easy

A. True Positives / (True Positives + False Positives)

B. 2 * (Precision * Recall) / (Precision + Recall)

C. (True Positives + True Negatives) / Total Predictions

D. True Positives / (True Positives + False Negatives)

Solution

1. Step 1: Recall the definition of precision

   Precision measures how many predicted positives are actually correct, so it is True Positives divided by all predicted positives.

2. Step 2: Match formula to definition

   True Positives + False Positives equals all predicted positives, so precision = TP / (TP + FP).

3. Final Answer:

   True Positives / (True Positives + False Positives) -> Option A

4. Quick Check:

   Precision = TP / (TP + FP) [OK]

Hint: Precision = correct positive predictions / all positive predictions [OK]

Common Mistakes:

• Using false negatives instead of false positives
• Confusing precision with recall formula
• Choosing accuracy formula by mistake

3. Given the confusion matrix:
TP=40, FP=10, FN=5, TN=45
What is the F1 score?

medium

A. 0.80

B. 0.88

C. 0.84

D. 0.90

Solution

1. Step 1: Calculate precision and recall

   Precision = TP / (TP + FP) = 40 / (40 + 10) = 40 / 50 = 0.8
   Recall = TP / (TP + FN) = 40 / (40 + 5) = 40 / 45 ≈ 0.8889

2. Step 2: Calculate F1 score using formula

   F1 = 2 * (Precision * Recall) / (Precision + Recall) = 2 * (0.8 * 0.8889) / (0.8 + 0.8889) ≈ 2 * 0.7111 / 1.6889 ≈ 1.4222 / 1.6889 ≈ 0.841

3. Final Answer:

   0.84 -> Option C

4. Quick Check:

   F1 balances precision and recall = 0.84 [OK]

Hint: F1 = 2 * (P * R) / (P + R), calculate P and R first [OK]

Common Mistakes:

• Mixing up false positives and false negatives
• Using accuracy instead of F1
• Rounding too early causing wrong final answer

4. You have this Python code to calculate recall:

def recall(tp, fn):
    return tp / (tp + fn + 1)

What is the error in this code?

medium

A. Adding 1 to denominator incorrectly lowers recall

B. Missing parentheses around tp + fn

C. Recall should use false positives, not false negatives

D. Function should return precision, not recall

Solution

1. Step 1: Understand recall formula

   Recall = TP / (TP + FN). The code adds +1 to denominator, which is not part of the formula.

2. Step 2: Identify effect of +1 in denominator

   Adding 1 artificially increases denominator, lowering recall value incorrectly.

3. Final Answer:

   Adding 1 to denominator incorrectly lowers recall -> Option A

4. Quick Check:

   Recall denominator = TP + FN only [OK]

Hint: Recall denominator must be TP + FN exactly, no extras [OK]

Common Mistakes:

• Thinking +1 is needed to avoid division by zero
• Confusing recall with precision formula
• Ignoring parentheses impact

5. A model has precision 0.75 and recall 0.60. You want to improve the F1 score. Which action is best?

hard

A. Increase precision by reducing false negatives

B. Increase recall by reducing false negatives

C. Increase accuracy by increasing true negatives

D. Increase recall by increasing false positives

Solution

1. Step 1: Understand F1 score depends on precision and recall

   F1 score balances precision and recall. Improving either precision or recall can increase F1.

2. Step 2: Identify how to improve recall

   Recall = TP / (TP + FN). Reducing false negatives (FN) increases recall.

3. Step 3: Evaluate options

   Increase recall by reducing false negatives increases recall by reducing FN, improving F1.
   Increase precision by reducing false negatives incorrectly links precision with false negatives.
   Increase accuracy by increasing true negatives affects accuracy, not F1 directly.
   Increase recall by increasing false positives, but increasing false positives lowers precision, hurting F1.

4. Final Answer:

   Increase recall by reducing false negatives -> Option B

5. Quick Check:

   Reduce FN to boost recall and F1 [OK]

Hint: Lower false negatives to raise recall and F1 score [OK]

Common Mistakes:

• Confusing false positives with false negatives
• Thinking accuracy always improves F1
• Increasing false positives to boost recall