Defining success criteria for agents in Agentic AI - Model Metrics & Evaluation

Metrics & Evaluation - Defining success criteria for agents

Which metric matters for this concept and WHY

When we want to know if an agent is successful, we need clear ways to measure it. Success criteria depend on what the agent is supposed to do. For example, if an agent answers questions, accuracy (how many answers are right) matters. If it completes tasks quickly, speed or efficiency matters. Sometimes, we combine several metrics like accuracy, speed, and user satisfaction to get a full picture. Choosing the right metric helps us know if the agent is doing a good job or needs improvement.

Confusion matrix or equivalent visualization (ASCII)

For agents that classify or decide, a confusion matrix helps us see how well they perform. It shows how many times the agent was right or wrong in different ways.

      Confusion Matrix:

          | Predicted Yes | Predicted No
      -----------------------------------
      Actual Yes |     TP       |     FN
      Actual No  |     FP       |     TN

      TP = True Positive (agent correct yes)
      FP = False Positive (agent wrong yes)
      TN = True Negative (agent correct no)
      FN = False Negative (agent wrong no)
    

This helps calculate precision, recall, and accuracy to understand success.

Precision vs Recall tradeoff with concrete examples

Imagine an agent that detects spam emails. If it marks too many good emails as spam (high false positives), users get annoyed. That means precision is low. If it misses many spam emails (high false negatives), spam floods inboxes, so recall is low.

We must balance precision and recall depending on what matters more. For spam, high precision is important to avoid losing good emails. For a medical agent detecting disease, high recall is key to catch all sick patients, even if some healthy ones get flagged.

What "good" vs "bad" metric values look like for this use case

Good success criteria mean the agent meets the goal well. For example:

• Accuracy above 90% for classification tasks.
• Precision and recall both above 85% for balanced detection tasks.
• Low task completion time for efficiency-focused agents.
• User satisfaction scores above 4 out of 5 for interactive agents.

Bad values are low accuracy (below 70%), big gaps between precision and recall, slow responses, or poor user feedback. These show the agent is not successful.

Metrics pitfalls

• Accuracy paradox: High accuracy can be misleading if data is unbalanced. For example, if 95% of emails are not spam, an agent that always says "not spam" has 95% accuracy but is useless.
• Data leakage: When the agent learns from information it should not have, making metrics look better than reality.
• Overfitting indicators: Very high training success but poor real-world results means the agent memorized data instead of learning general rules.
• Ignoring context: Using wrong metrics for the task can hide problems. For example, using accuracy alone for rare event detection.

Self-check question

Your agent has 98% accuracy but only 12% recall on detecting fraud. Is it good for production? Why or why not?

Answer: No, it is not good. The low recall means the agent misses most fraud cases, which is very risky. Even though accuracy is high, it mostly predicts "no fraud" correctly because fraud is rare. For fraud detection, catching as many fraud cases as possible (high recall) is more important.

Key Result

Choosing the right success metric depends on the agent's goal; balancing precision and recall is key for reliable performance.

Practice

1. Why is it important to define success criteria for an AI agent?

easy

A. It reduces the size of the agent's code.

B. It helps the agent understand what goal to achieve.

C. It makes the agent run faster.

D. It allows the agent to ignore errors.

Solution

1. Step 1: Understand the role of success criteria

   Success criteria tell the agent what outcome is desired or considered good.

2. Step 2: Connect success criteria to agent behavior

   Without clear goals, the agent cannot know what to aim for or when it has succeeded.

3. Final Answer:

   It helps the agent understand what goal to achieve. -> Option B

4. Quick Check:

   Success criteria = clear goals [OK]

Hint: Success criteria define the agent's goal clearly [OK]

Common Mistakes:

• Thinking success criteria speed up the agent
• Confusing success criteria with code size
• Believing success criteria ignore errors

2. Which of the following is the correct way to express a success criterion for an agent in code?

easy

A. success == accuracy > 0.9

B. success = accuracy = 0.9

C. success = accuracy > 0.9

D. success => accuracy > 0.9

Solution

1. Step 1: Identify correct comparison syntax

   In Python, to assign a boolean result, use a single = with a comparison expression on the right.

2. Step 2: Check each option's syntax

   success = accuracy > 0.9 uses correct assignment and comparison. success = accuracy = 0.9 uses = instead of == for comparison. success == accuracy > 0.9 uses == incorrectly for assignment. success => accuracy > 0.9 uses => which is invalid in Python.

3. Final Answer:

   success = accuracy > 0.9 -> Option C

4. Quick Check:

   Assignment with comparison uses = and > [OK]

Hint: Use '=' for assignment, '>' for comparison [OK]

Common Mistakes:

• Using '==' instead of '=' for assignment
• Using '=' instead of '==' for comparison
• Using invalid operators like '=>'

3. Given the code below, what will be the value of success?

accuracy = 0.85
threshold = 0.8
success = accuracy >= threshold

medium

A. True

B. Error

C. 0.85

D. False

Solution

1. Step 1: Compare accuracy and threshold values

   Accuracy is 0.85, threshold is 0.8, so 0.85 >= 0.8 is True.

2. Step 2: Assign comparison result to success

   The boolean True is assigned to success.

3. Final Answer:

   True -> Option A

4. Quick Check:

   0.85 >= 0.8 = True [OK]

Hint: Check if accuracy meets or exceeds threshold [OK]

Common Mistakes:

• Confusing value 0.85 with boolean True
• Thinking comparison returns a number
• Expecting an error from valid comparison

4. The following code is intended to check if an agent's success metric is above 90%, but it has a bug. What is the bug?

success_metric = 0.92
if success_metric = 0.9:
    print('Agent succeeded')

medium

A. Missing colon ':' after if statement

B. Print statement syntax error

C. Incorrect variable name 'success_metric'

D. Using '=' instead of '==' in the if condition

Solution

1. Step 1: Identify the if statement syntax

   In Python, '=' is for assignment, '==' is for comparison in conditions.

2. Step 2: Locate the bug in the if condition

   The code uses '=' instead of '==' which causes a syntax error.

3. Final Answer:

   Using '=' instead of '==' in the if condition -> Option D

4. Quick Check:

   Use '==' for comparison in if [OK]

Hint: Use '==' to compare values in if statements [OK]

Common Mistakes:

• Confusing '=' with '==' in conditions
• Ignoring syntax errors from wrong operators
• Assuming missing colon is the error

5. You want to define success criteria for an agent that completes tasks with at least 95% accuracy and finishes within 10 seconds. Which of the following is the best way to define this success criteria in code?

hard

A. success = (accuracy >= 0.95) and (time_taken <= 10)

B. success = accuracy > 0.95 or time_taken < 10

C. success = accuracy == 0.95 and time_taken == 10

D. success = accuracy >= 0.95 and time_taken > 10

Solution

1. Step 1: Understand the criteria requirements

   The agent must have accuracy at least 95% and finish within 10 seconds.

2. Step 2: Translate criteria into logical conditions

   Use '>=' for accuracy and '<=' for time, combined with 'and' to require both.

3. Step 3: Evaluate each option

   success = (accuracy >= 0.95) and (time_taken <= 10) correctly uses 'and' and proper comparisons. success = accuracy > 0.95 or time_taken < 10 uses 'or' which allows passing if only one condition is met. success = accuracy == 0.95 and time_taken == 10 uses '==' which is too strict. success = accuracy >= 0.95 and time_taken > 10 allows time_taken > 10 which breaks the time limit.

4. Final Answer:

   success = (accuracy >= 0.95) and (time_taken <= 10) -> Option A

5. Quick Check:

   Both accuracy and time must meet thresholds [OK]

Hint: Use 'and' to combine all success conditions [OK]

Common Mistakes:

• Using 'or' instead of 'and' to combine conditions
• Using '==' instead of '>=' or '<='
• Allowing time greater than limit