Measuring agent accuracy and relevance in Agentic AI

Introduction

We measure accuracy and relevance to know how well an AI agent answers questions or solves tasks. This helps us trust and improve the agent.

Checking if a chatbot gives correct answers to customer questions.

Testing if a recommendation agent suggests useful products.

Evaluating if a virtual assistant understands and completes commands properly.

Comparing different AI agents to pick the best one for a job.

Syntax

Agentic AI

accuracy = (number of correct answers) / (total answers)
relevance = (number of relevant answers) / (total answers)

Accuracy measures how many answers are exactly right.

Relevance measures how many answers are useful or related, even if not perfect.

Examples

This shows accuracy and relevance as simple fractions.

Agentic AI

accuracy = 8 / 10  # 8 correct out of 10 answers
relevance = 9 / 10 # 9 relevant out of 10 answers

Use variables to calculate accuracy and relevance in code.

Agentic AI

accuracy = correct_predictions / total_predictions
relevance = relevant_responses / total_responses

Sample Model

This program calculates and prints accuracy and relevance for an AI agent's answers.

Agentic AI

correct_answers = 7
relevant_answers = 9
total_answers = 10

accuracy = correct_answers / total_answers
relevance = relevant_answers / total_answers

print(f"Accuracy: {accuracy:.2f}")
print(f"Relevance: {relevance:.2f}")

Important Notes

Accuracy is strict; only fully correct answers count.

Relevance allows some flexibility; answers can be helpful even if not perfect.

Always check both to get a full picture of agent performance.

Summary

Accuracy shows how many answers are exactly right.

Relevance shows how many answers are useful or related.

Measuring both helps improve and trust AI agents.