Error rate and failure analysis in Agentic Ai

Introduction

Error rate tells us how often a model makes mistakes. Failure analysis helps find why and where these mistakes happen.

Checking how well a spam filter catches unwanted emails

Finding why a voice assistant misunderstands commands

Improving a self-driving car's object detection errors

Evaluating a medical test's wrong diagnosis cases

Understanding mistakes in a recommendation system

Syntax

Agentic_ai

error_rate = number_of_wrong_predictions / total_predictions

# For failure analysis, review wrong cases and find patterns

Error rate is a simple fraction showing mistakes over total tries.

Failure analysis is more about looking closely at errors to improve the model.

Examples

This means the model is wrong 5% of the time.

Agentic_ai

error_rate = 5 / 100  # 5 wrong out of 100 predictions

Collecting wrong predictions helps us see patterns causing errors.

Agentic_ai

wrong_cases = [case for case in test_data if model.predict(case) != case.label]
# Look at wrong_cases to find common issues

Sample Program

This code trains a simple decision tree on iris data, calculates error rate, and prints some wrong predictions for failure analysis.

Agentic_ai

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import accuracy_score

# Load data
iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(iris.data, iris.target, test_size=0.3, random_state=42)

# Train model
model = DecisionTreeClassifier(random_state=42)
model.fit(X_train, y_train)

# Predict
predictions = model.predict(X_test)

# Calculate error rate
wrong = sum(predictions != y_test)
total = len(y_test)
error_rate = wrong / total

print(f"Wrong predictions: {wrong}")
print(f"Total predictions: {total}")
print(f"Error rate: {error_rate:.2f}")

# Failure analysis: show some wrong cases
error_count = 0
for i, (pred, true) in enumerate(zip(predictions, y_test)):
    if pred != true:
        print(f"Index {i}: predicted {pred}, actual {true}")
        error_count += 1
        if error_count >= 3:  # show only first 3 errors
            break

Important Notes

Always check error rate on new data to see real performance.

Failure analysis helps find if errors happen on specific groups or conditions.

Reducing error rate improves trust in your model.

Summary

Error rate shows how often a model is wrong.

Failure analysis looks closely at mistakes to understand causes.

Both help improve machine learning models step by step.