Supervisor agent pattern in Agentic AI - Model Metrics & Evaluation

Metrics & Evaluation - Supervisor agent pattern

Which metric matters for the Supervisor agent pattern and WHY

The Supervisor agent pattern involves a main agent overseeing other agents to ensure correct task completion. The key metric here is accuracy of the supervisor's decisions to accept or reject sub-agent outputs. This accuracy ensures the supervisor correctly identifies good or bad outputs, maintaining overall system reliability.

Additionally, precision and recall are important to measure how well the supervisor balances catching errors (recall) without wrongly rejecting good outputs (precision).

Confusion matrix for Supervisor agent decisions

      | Predicted Good       | Predicted Bad        |
      |---------------------|----------------------|
      | True Positive  (TP)  | False Negative (FN)  |
      | False Positive (FP)  | True Negative  (TN)  |

      TP: Supervisor correctly accepts good output
      FP: Supervisor wrongly accepts bad output
      FN: Supervisor wrongly rejects good output
      TN: Supervisor correctly rejects bad output
    

Precision vs Recall tradeoff in Supervisor agent pattern

Precision measures how many accepted outputs are truly good. High precision means the supervisor rarely accepts bad outputs, avoiding errors downstream.

Recall measures how many good outputs the supervisor correctly accepts. High recall means the supervisor rarely rejects good outputs, avoiding unnecessary rework.

Tradeoff example: If the supervisor is too strict, recall drops (many good outputs rejected), causing delays. If too lenient, precision drops (bad outputs accepted), causing errors.

What "good" vs "bad" metric values look like for Supervisor agent pattern

• Good: Accuracy > 90%, Precision > 85%, Recall > 85% - Supervisor reliably accepts good outputs and rejects bad ones.
• Bad: Accuracy < 70%, Precision < 60%, Recall < 60% - Supervisor often makes wrong decisions, harming system trust.

Common pitfalls in evaluating Supervisor agent pattern metrics

• Accuracy paradox: If bad outputs are rare, high accuracy can be misleading if supervisor always accepts outputs.
• Data leakage: Using future or test data to train supervisor inflates metrics falsely.
• Overfitting: Supervisor tuned too closely to training data may fail on new outputs.

Self-check question

Your supervisor agent has 98% accuracy but only 12% recall on bad outputs. Is it good for production? Why or why not?

Answer: No, it is not good. The supervisor misses 88% of bad outputs (low recall), allowing many errors through. High accuracy is misleading because bad outputs are rare. Improving recall is critical to catch errors.

Key Result

Supervisor agent pattern needs balanced precision and recall to reliably accept good outputs and reject bad ones.

Practice

1. What is the main role of a Supervisor agent in the supervisor agent pattern?

easy

A. To collect raw data from sensors

B. To train a single AI model

C. To replace all other agents with one

D. To manage and coordinate multiple AI agents

Solution

1. Step 1: Understand the supervisor agent's purpose

   The supervisor agent is designed to oversee and coordinate multiple AI agents working together.

2. Step 2: Differentiate from other roles

   Unlike training or data collection, the supervisor agent focuses on managing teamwork and quality control.

3. Final Answer:

   To manage and coordinate multiple AI agents -> Option D

4. Quick Check:

   Supervisor agent = manager of multiple agents [OK]

Hint: Supervisor agent = team manager of AI agents [OK]

Common Mistakes:

• Confusing supervisor with data collector
• Thinking supervisor trains models directly
• Assuming supervisor replaces all agents

2. Which of the following is the correct way to describe the supervisor agent's function in code?

easy

A. supervisor.replace_agents()

B. supervisor.train_single_model(data)

C. supervisor.collect_results(agents)

D. supervisor.ignore_agent_outputs()

Solution

1. Step 1: Identify supervisor's interaction with agents

   The supervisor collects and evaluates results from multiple agents, so a method like collect_results fits.

2. Step 2: Eliminate incorrect options

   Training a single model, replacing agents, or ignoring outputs do not match the supervisor's coordination role.

3. Final Answer:

   supervisor.collect_results(agents) -> Option C

4. Quick Check:

   Supervisor collects results = collect_results() [OK]

Hint: Supervisor collects and evaluates agent outputs [OK]

Common Mistakes:

• Choosing training method instead of collection
• Thinking supervisor replaces agents
• Ignoring outputs contradicts supervisor role

3. Given this code snippet for a supervisor agent pattern, what will be the printed output?

class Agent:
    def __init__(self, name, score):
        self.name = name
        self.score = score

class Supervisor:
    def __init__(self, agents):
        self.agents = agents
    def best_agent(self):
        return max(self.agents, key=lambda a: a.score).name

agents = [Agent('A1', 85), Agent('A2', 90), Agent('A3', 88)]
supervisor = Supervisor(agents)
print(supervisor.best_agent())

medium

A. A1

B. A2

C. A3

D. None

Solution

1. Step 1: Understand the agent scores

   Agents have scores: A1=85, A2=90, A3=88.

2. Step 2: Identify the agent with the highest score

   The best_agent method returns the name of the agent with the max score, which is A2 with 90.

3. Final Answer:

   A2 -> Option B

4. Quick Check:

   Max score agent = A2 [OK]

Hint: Max score agent name is printed [OK]

Common Mistakes:

• Choosing agent with second highest score
• Confusing agent names
• Assuming None if not found

4. Identify the bug in this supervisor agent code snippet:

class Supervisor:
    def __init__(self, agents):
        self.agents = agents
    def best_score(self):
        return max(self.agents, key=lambda a: a.score)

agents = [{'name': 'A1', 'score': 80}, {'name': 'A2', 'score': 95}]
supervisor = Supervisor(agents)
print(supervisor.best_score())

medium

A. Agents should be objects, not dictionaries

B. max() function is used incorrectly

C. Missing return statement in best_score

D. Supervisor class missing __init__ method

Solution

1. Step 1: Check agent data type and usage

   The best_score method expects agents with attribute score, but agents are dictionaries, not objects.

2. Step 2: Understand attribute vs key access

   Using a.score on a dictionary causes an error; dictionaries need a['score'].

3. Final Answer:

   Agents should be objects, not dictionaries -> Option A

4. Quick Check:

   Attribute access on dict causes error [OK]

Hint: Use objects or adjust attribute access for dicts [OK]

Common Mistakes:

• Thinking max() usage is wrong
• Missing return statement (it exists)
• Ignoring data type mismatch

5. You want to design a supervisor agent that combines outputs from three different AI agents solving a complex task. Which approach best fits the supervisor agent pattern?

hard

A. Collect outputs, evaluate quality, and select the best result

B. Run only the fastest agent and ignore others

C. Train all agents on the same data independently

D. Replace all agents with a single large model

Solution

1. Step 1: Understand supervisor agent's coordination role

   The supervisor should gather outputs from all agents and decide which is best based on quality.

2. Step 2: Evaluate other options

   Ignoring agents, training independently without coordination, or replacing agents contradict the supervisor pattern.

3. Final Answer:

   Collect outputs, evaluate quality, and select the best result -> Option A

4. Quick Check:

   Supervisor = collect + evaluate + select best [OK]

Hint: Supervisor picks best output from all agents [OK]

Common Mistakes:

• Ignoring some agents' outputs
• Confusing training with supervising
• Replacing agents instead of coordinating