Agent API design patterns in Agentic AI - Model Metrics & Evaluation

Metrics & Evaluation - Agent API design patterns

Which metric matters for Agent API design patterns and WHY

When designing Agent APIs, key metrics focus on response accuracy, latency, and robustness. Accuracy measures if the agent returns correct and relevant answers. Latency checks how fast the agent responds, important for user experience. Robustness ensures the agent handles unexpected inputs without failure. These metrics matter because a good API must be reliable, fast, and correct to be useful in real-world applications.

Confusion matrix or equivalent visualization

    | Predicted Correct | Predicted Incorrect |
    |-------------------|---------------------|
    | True Positive (TP) | False Positive (FP)  |
    | False Negative (FN)| True Negative (TN)   |

    Example:
    TP = 80 (correct responses)
    FP = 10 (incorrect but accepted)
    FN = 5  (missed correct responses)
    TN = 5  (correctly rejected wrong inputs)

    Total samples = 100
    

This matrix helps measure precision and recall of the agent's responses.

Precision vs Recall tradeoff with concrete examples

Precision means how many responses the agent gave that were actually correct. High precision means fewer wrong answers.

Recall means how many of all possible correct answers the agent found. High recall means the agent misses fewer correct answers.

For example, a customer support agent API should have high precision to avoid giving wrong advice. But a research assistant agent API should have high recall to find as many relevant facts as possible, even if some are less precise.

What "good" vs "bad" metric values look like for Agent API design

• Good: Precision > 0.9, Recall > 0.85, Latency < 1 second, Robustness handles 99% of unexpected inputs without failure.
• Bad: Precision < 0.6 (many wrong answers), Recall < 0.5 (misses many correct answers), Latency > 5 seconds (slow response), Frequent crashes or errors on unusual inputs.

Common pitfalls in Agent API metrics

• Accuracy paradox: High overall accuracy can hide poor performance on rare but important queries.
• Data leakage: Training on data too similar to test data inflates metrics falsely.
• Overfitting: Agent performs well on training queries but poorly on new, real-world inputs.
• Ignoring latency: A very accurate agent that responds too slowly harms user experience.
• Not measuring robustness: Failing to test how the agent handles unexpected or malformed inputs.

Self-check question

Your agent API has 98% accuracy but only 12% recall on critical queries. Is it good for production? Why or why not?

Answer: No, it is not good. Although accuracy is high, the very low recall means the agent misses most important queries. This can cause serious problems because many correct answers are never found. Improving recall is critical before production use.

Key Result

For Agent API design, balancing high precision, recall, low latency, and robustness ensures reliable and useful agent responses.

Practice

1. What is the main purpose of using Agent API design patterns in AI systems?

easy

A. To organize how AI agents communicate and work together

B. To speed up the training of machine learning models

C. To store large datasets efficiently

D. To improve the hardware performance of AI servers

Solution

1. Step 1: Understand the role of Agent API design patterns

   These patterns help define clear communication and interaction rules between AI agents.

2. Step 2: Compare with other options

   Options A, C, and D relate to training speed, data storage, and hardware, which are not the focus of Agent API design patterns.

3. Final Answer:

   To organize how AI agents communicate and work together -> Option A

4. Quick Check:

   Agent API design patterns = organize communication [OK]

Hint: Agent API patterns focus on agent communication, not hardware or data [OK]

Common Mistakes:

• Confusing design patterns with hardware optimization
• Thinking patterns speed up model training directly
• Mixing data storage with agent communication

2. Which of the following is the correct way to define a simple message passing function in an Agent API?

easy

A. def send_message(agent, message): return message + agent

B. def send_message(agent, message): agent.send(message)

C. def send_message(agent, message): return agent.receive(message)

D. def send_message(agent, message): print(agent + message)

Solution

1. Step 1: Analyze the function purpose

   The function should send a message to an agent and get a response by calling the agent's receive method.

2. Step 2: Check each option

   def send_message(agent, message): return agent.receive(message) correctly calls agent.receive(message). def send_message(agent, message): agent.send(message) calls agent.send which is not standard. Options A and C incorrectly try to add or print agent and message.

3. Final Answer:

   def send_message(agent, message): return agent.receive(message) -> Option C

4. Quick Check:

   Message passing calls agent.receive(message) [OK]

Hint: Message passing calls agent.receive(message) to send data [OK]

Common Mistakes:

• Using agent.send instead of agent.receive
• Trying to concatenate agent object with string
• Printing instead of returning the message

3. Given the code below, what will be the output?

class Agent:
    def receive(self, message):
        return f"Received: {message}"

def send_message(agent, message):
    return agent.receive(message)

agent = Agent()
print(send_message(agent, "Hello"))

medium

A. Error: method not found

B. Hello

C. send_message(agent, Hello)

D. Received: Hello

Solution

1. Step 1: Understand the Agent class and receive method

   The receive method returns the string 'Received: ' plus the message passed.

2. Step 2: Trace the send_message call

   send_message calls agent.receive with "Hello", so it returns 'Received: Hello'.

3. Final Answer:

   Received: Hello -> Option D

4. Quick Check:

   agent.receive("Hello") = "Received: Hello" [OK]

Hint: Agent.receive returns 'Received: ' plus message [OK]

Common Mistakes:

• Expecting just the message without prefix
• Thinking send_message prints instead of returns
• Assuming method does not exist causing error

4. Identify the error in the following Agent API code snippet:

class Agent:
    def receive(self, message):
        print(f"Got message: {message}")

def send_message(agent, message):
    return agent.receive(message)

agent = Agent()
response = send_message(agent, "Hi")
print(response)

medium

A. The receive method should return a value, not just print

B. send_message should not call agent.receive

C. Agent class is missing an __init__ method

D. The print statement in send_message is incorrect

Solution

1. Step 1: Check receive method behavior

   receive only prints the message but does not return anything, so it returns None by default.

2. Step 2: Analyze send_message and print(response)

   send_message returns None, so printing response outputs None, which is likely unintended.

3. Final Answer:

   The receive method should return a value, not just print -> Option A

4. Quick Check:

   receive must return message for send_message to work [OK]

Hint: receive must return, not just print, to pass data back [OK]

Common Mistakes:

• Ignoring that print returns None
• Thinking __init__ is required here
• Confusing print location with syntax error

5. You want to design an Agent API where multiple agents can collaborate by passing messages and roles define their behavior. Which design pattern best supports this?

hard

A. Factory pattern to create agents dynamically

B. Mediator pattern to centralize communication between agents

C. Singleton pattern to ensure one agent instance

D. Observer pattern to notify agents of state changes

Solution

1. Step 1: Understand collaboration and role-based behavior

   Agents need a central way to communicate and coordinate roles effectively.

2. Step 2: Match design patterns to needs

   The Mediator pattern centralizes communication, making it ideal for agent collaboration. Singleton limits to one instance, Factory creates objects, Observer handles notifications but not central communication.

3. Final Answer:

   Mediator pattern to centralize communication between agents -> Option B

4. Quick Check:

   Collaboration with roles = Mediator pattern [OK]

Hint: Mediator centralizes agent communication for collaboration [OK]

Common Mistakes:

• Choosing Singleton which limits to one agent
• Confusing Factory with communication pattern
• Using Observer which is for event notification only