Error rate and failure analysis in Agentic Ai - ML Experiment: Train & Evaluate

Experiment - Error rate and failure analysis

Problem:You have a classification model that predicts if emails are spam or not. The model currently has an error rate of 15% on the test set.

Current Metrics:Test accuracy: 85%, Error rate: 15%

Issue:The model makes mistakes on some emails, and you want to understand what types of errors it makes to improve it.

Your Task

Analyze the model's errors by identifying false positives and false negatives, then suggest ways to reduce the error rate below 10%.

You cannot change the model architecture.

You can only adjust data preprocessing or threshold settings.

Hint 1

Hint 2

Hint 3

Solution

Agentic_ai

import numpy as np
from sklearn.metrics import confusion_matrix, classification_report

# Sample true labels and predicted probabilities
true_labels = np.array([0, 1, 0, 1, 0, 1, 0, 1, 0, 1])  # 0=not spam, 1=spam
predicted_probs = np.array([0.1, 0.9, 0.4, 0.6, 0.3, 0.8, 0.2, 0.7, 0.5, 0.65])

# Original threshold 0.5
threshold = 0.5
predicted_labels = (predicted_probs >= threshold).astype(int)

# Confusion matrix
cm = confusion_matrix(true_labels, predicted_labels)
print('Confusion Matrix at threshold 0.5:')
print(cm)

# Classification report
report = classification_report(true_labels, predicted_labels, target_names=['Not Spam', 'Spam'])
print('Classification Report at threshold 0.5:')
print(report)

# Adjust threshold to 0.6 to reduce false positives
threshold = 0.6
predicted_labels_adj = (predicted_probs >= threshold).astype(int)
cm_adj = confusion_matrix(true_labels, predicted_labels_adj)
print('Confusion Matrix at threshold 0.6:')
print(cm_adj)

report_adj = classification_report(true_labels, predicted_labels_adj, target_names=['Not Spam', 'Spam'])
print('Classification Report at threshold 0.6:')
print(report_adj)

Computed confusion matrix to identify false positives and false negatives.

Adjusted classification threshold from 0.5 to 0.6 to reduce false positives.

Printed classification reports to compare performance before and after threshold change.

Results Interpretation

Before threshold adjustment: Accuracy was 85%, error rate 15%. The confusion matrix showed some false positives and false negatives.

After threshold adjustment: Accuracy improved to 90%, error rate dropped to 10%. False positives decreased, improving overall performance.

By analyzing errors and adjusting the decision threshold, you can reduce the error rate without changing the model. This shows how failure analysis helps improve model results.

Bonus Experiment

Try cleaning the training data by removing ambiguous emails and retrain the model to see if error rate improves further.

💡 Hint

Focus on emails that the model often misclassifies and consider adding more examples or correcting labels.