Measuring agent accuracy and relevance in Agentic AI - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Metrics

intermediate

Understanding Precision and Recall in Agent Responses

An AI agent answers 100 questions. It correctly answers 80 questions but also gives 20 wrong answers. What is the precision of the agent?

A 0.8

B 0.5

C 0.67

D 1.0

💡 Hint

Precision is the number of correct answers divided by total answers given.

Precision = Correct answers / (Correct + Wrong answers) = 80 / (80 + 20) = 0.8

🧠 Conceptual

intermediate

Difference Between Accuracy and Relevance

Which statement best describes the difference between accuracy and relevance when measuring an AI agent's output?

A Accuracy and relevance mean the same thing in AI agent evaluation.

B Accuracy measures speed; relevance measures correctness.

C Accuracy measures correctness; relevance measures how useful the answer is to the question.

D Accuracy measures how often the agent responds; relevance measures the length of the response.

💡 Hint

Think about correctness versus usefulness.

Accuracy is about correct answers. Relevance is about how well the answer fits the question's intent.

❓ Predict Output

advanced

Calculating F1 Score for Agent Responses

Given the following code calculating precision and recall, what is the printed F1 score?

Agentic AI

correct = 70
wrong = 30
relevant = 80
precision = correct / (correct + wrong)
recall = correct / relevant
f1_score = 2 * (precision * recall) / (precision + recall)
print(round(f1_score, 2))

A 0.75

B 0.85

C 0.82

D 0.78

💡 Hint

Calculate precision and recall first, then use the F1 formula.

Precision = 70/(70+30) = 0.7; Recall = 70/80 = 0.875; F1 = 2*(0.7*0.875)/(0.7+0.875) ≈ 0.78

🔧 Debug

advanced

Identifying the Error in Agent Accuracy Calculation

What error does the following code produce when calculating accuracy?

correct = 50
wrong = 20
accuracy = correct / wrong
print(accuracy)

Agentic AI

correct = 50
wrong = 20
accuracy = correct / wrong
print(accuracy)

A It raises a ZeroDivisionError.

B It prints 2.5, which is incorrect for accuracy calculation.

C It prints 0.714, which is the correct accuracy.

D It raises a TypeError.

💡 Hint

Think about how accuracy is calculated.

Accuracy should be correct / (correct + wrong), but code divides correct by wrong, giving 2.5 which is not a valid accuracy.

❓ Model Choice

expert

Choosing the Best Metric for Evaluating Agent Relevance

You want to measure how well an AI agent's answers match the user's intent, focusing on usefulness rather than just correctness. Which metric is best suited for this?

A Mean Reciprocal Rank (MRR)

B Precision

C BLEU score

D Accuracy

💡 Hint

Consider metrics that evaluate ranking or relevance rather than exact correctness.

MRR measures how high relevant answers appear in a ranked list, capturing usefulness and relevance better than simple accuracy or precision.

Practice

1. What does accuracy measure when evaluating an AI agent's answers?

easy

A. How many answers are related but not exact

B. How fast the agent responds

C. How many answers are exactly correct

D. How many answers are generated

Solution

1. Step 1: Understand accuracy definition

   Accuracy counts the number of answers that match the correct ones exactly.

2. Step 2: Compare with other metrics

   Relevance measures usefulness, not exact correctness, so it is different from accuracy.

3. Final Answer:

   How many answers are exactly correct -> Option C

4. Quick Check:

   Accuracy = exact correctness [OK]

Hint: Accuracy means exact right answers only [OK]

Common Mistakes:

• Confusing accuracy with relevance
• Thinking accuracy measures speed
• Assuming accuracy counts all related answers

2. Which of the following is the correct way to calculate accuracy for an AI agent's answers?

easy

A. Number of related answers divided by total answers

B. Number of correct answers divided by total answers

C. Number of answers generated per second

D. Number of answers ignored by the agent

Solution

1. Step 1: Recall accuracy formula

   Accuracy = (correct answers) / (total answers given).

2. Step 2: Eliminate incorrect options

   Options about related answers or speed do not define accuracy.

3. Final Answer:

   Number of correct answers divided by total answers -> Option B

4. Quick Check:

   Accuracy = correct / total [OK]

Hint: Accuracy = correct answers ÷ total answers [OK]

Common Mistakes:

• Using related answers count instead of correct
• Mixing speed with accuracy
• Ignoring total number of answers

3. Given an AI agent answered 80 questions, 60 were exactly correct, and 10 more were relevant but not exact. What is the accuracy and relevance percentage?

medium

A. Accuracy 60%, Relevance 70%

B. Accuracy 60%, Relevance 87.5%

C. Accuracy 75%, Relevance 60%

D. Accuracy 75%, Relevance 87.5%

Solution

1. Step 1: Calculate accuracy percentage

   Accuracy = (60 correct / 80 total) * 100 = 75%.

2. Step 2: Calculate relevance percentage

   Relevance = ((60 correct + 10 relevant) / 80 total) * 100 = 87.5%.

3. Final Answer:

   Accuracy 75%, Relevance 87.5% -> Option D

4. Quick Check:

   Accuracy = 75%, Relevance = 87.5% [OK]

Hint: Add relevant to correct for relevance % [OK]

Common Mistakes:

• Mixing accuracy and relevance values
• Not adding relevant answers for relevance
• Dividing by wrong total number

4. An AI agent evaluation code snippet is below. It calculates accuracy but returns 0. What is the bug?

correct = 50
total = 0
accuracy = correct / total
print(accuracy)

medium

A. Division by zero error due to total being zero

B. Correct variable is zero, so accuracy is zero

C. Print statement syntax is wrong

D. Accuracy should be multiplied by 100

Solution

1. Step 1: Identify variables and operation

   correct = 50, total = 0, accuracy = correct / total.

2. Step 2: Check for division errors

   Dividing by zero (total=0) causes an error or invalid result.

3. Final Answer:

   Division by zero error due to total being zero -> Option A

4. Quick Check:

   Division by zero causes error [OK]

Hint: Check denominator is not zero before dividing [OK]

Common Mistakes:

• Ignoring zero division error
• Thinking print syntax is wrong
• Assuming accuracy must be multiplied by 100

5. You want to improve an AI agent's trust by measuring both accuracy and relevance. Which approach best helps achieve this?

hard

A. Track exact correct answers and also count useful related answers

B. Only count answers that are exactly correct

C. Ignore relevance and focus on speed of answers

D. Count all answers regardless of correctness or relevance

Solution

1. Step 1: Understand trust factors

   Trust improves when answers are both correct and useful (relevant).

2. Step 2: Choose measurement approach

   Tracking both exact correctness (accuracy) and usefulness (relevance) gives a fuller picture.

3. Final Answer:

   Track exact correct answers and also count useful related answers -> Option A

4. Quick Check:

   Measure accuracy + relevance for trust [OK]

Hint: Measure both exact and useful answers for trust [OK]

Common Mistakes:

• Focusing only on exact correctness
• Ignoring relevance completely
• Measuring speed instead of quality