Measuring agent accuracy and relevance in Agentic AI - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Metrics

intermediate

Understanding Precision and Recall in Agent Responses

An AI agent answers 100 questions. It correctly answers 80 questions but also gives 20 wrong answers. What is the precision of the agent?

A 0.8

B 0.5

C 0.67

D 1.0

💡 Hint

Precision is the number of correct answers divided by total answers given.

Precision = Correct answers / (Correct + Wrong answers) = 80 / (80 + 20) = 0.8

🧠 Conceptual

intermediate

Difference Between Accuracy and Relevance

Which statement best describes the difference between accuracy and relevance when measuring an AI agent's output?

A Accuracy and relevance mean the same thing in AI agent evaluation.

B Accuracy measures speed; relevance measures correctness.

C Accuracy measures correctness; relevance measures how useful the answer is to the question.

D Accuracy measures how often the agent responds; relevance measures the length of the response.

💡 Hint

Think about correctness versus usefulness.

Accuracy is about correct answers. Relevance is about how well the answer fits the question's intent.

❓ Predict Output

advanced

Calculating F1 Score for Agent Responses

Given the following code calculating precision and recall, what is the printed F1 score?

Agentic AI

correct = 70
wrong = 30
relevant = 80
precision = correct / (correct + wrong)
recall = correct / relevant
f1_score = 2 * (precision * recall) / (precision + recall)
print(round(f1_score, 2))

A 0.75

B 0.85

C 0.82

D 0.78

💡 Hint

Calculate precision and recall first, then use the F1 formula.

Precision = 70/(70+30) = 0.7; Recall = 70/80 = 0.875; F1 = 2*(0.7*0.875)/(0.7+0.875) ≈ 0.78

🔧 Debug

advanced

Identifying the Error in Agent Accuracy Calculation

What error does the following code produce when calculating accuracy?

correct = 50
wrong = 20
accuracy = correct / wrong
print(accuracy)

Agentic AI

correct = 50
wrong = 20
accuracy = correct / wrong
print(accuracy)

A It raises a ZeroDivisionError.

B It prints 2.5, which is incorrect for accuracy calculation.

C It prints 0.714, which is the correct accuracy.

D It raises a TypeError.

💡 Hint

Think about how accuracy is calculated.

Accuracy should be correct / (correct + wrong), but code divides correct by wrong, giving 2.5 which is not a valid accuracy.

❓ Model Choice

expert

Choosing the Best Metric for Evaluating Agent Relevance

You want to measure how well an AI agent's answers match the user's intent, focusing on usefulness rather than just correctness. Which metric is best suited for this?

A Mean Reciprocal Rank (MRR)

B Precision

C BLEU score

D Accuracy

💡 Hint

Consider metrics that evaluate ranking or relevance rather than exact correctness.

MRR measures how high relevant answers appear in a ranked list, capturing usefulness and relevance better than simple accuracy or precision.