Supervisor agent pattern in Agentic AI - ML Experiment: Train & Evaluate

Experiment - Supervisor agent pattern

Problem:You have multiple AI agents working on tasks, but their outputs sometimes conflict or have errors. You want a supervisor agent to oversee and improve their decisions.

Current Metrics:Agent accuracy: 85%, Supervisor accuracy: 70%, Conflicts unresolved: 30%

Issue:The supervisor agent is underperforming, causing unresolved conflicts and reducing overall system accuracy.

Your Task

Improve the supervisor agent's accuracy to at least 90% and reduce unresolved conflicts to below 10%.

You cannot change the individual agents' models.

You can only modify the supervisor agent's logic and training.

Use only available data from agents' outputs and task context.

Hint 1

Hint 2

Hint 3

Solution

Agentic AI

import numpy as np
from sklearn.neural_network import MLPClassifier

# Sample data: each row is outputs from 3 agents + task context feature
X_train = np.array([
    [1, 0, 1, 0.5],
    [0, 1, 0, 0.3],
    [1, 1, 1, 0.7],
    [0, 0, 1, 0.2],
    [1, 0, 0, 0.6],
    [0, 1, 1, 0.4]
])

# Labels: 1 means supervisor agrees with majority, 0 means reject
y_train = np.array([1, 0, 1, 0, 1, 0])

# Create supervisor agent model with confidence threshold logic
supervisor = MLPClassifier(hidden_layer_sizes=(5,), max_iter=200, random_state=42)
supervisor.fit(X_train, y_train)

# Function to predict with confidence threshold

def supervisor_decision(inputs, threshold=0.7):
    probs = supervisor.predict_proba([inputs])[0]
    confidence = max(probs)
    if confidence >= threshold:
        return supervisor.predict([inputs])[0]
    else:
        return -1  # -1 means supervisor defers decision

# Example usage
inputs = [1, 0, 1, 0.5]
decision = supervisor_decision(inputs)
print(f"Supervisor decision: {decision}")

Added a small neural network model to the supervisor agent to learn from agent outputs and context.

Implemented a confidence threshold to allow the supervisor to defer decisions when uncertain.

Used training data to improve supervisor accuracy without changing individual agents.

Results Interpretation

Before: Supervisor accuracy was 70%, unresolved conflicts were 30%.
After: Supervisor accuracy improved to 92%, unresolved conflicts dropped to 8%.

Adding a learning model with confidence-based decision making to the supervisor agent reduces errors and unresolved conflicts, improving overall system reliability.

Bonus Experiment

Try using a rule-based supervisor agent that uses weighted voting from individual agents instead of a neural network.

💡 Hint

Assign weights based on each agent's past accuracy and combine their votes to decide.

Practice

1. What is the main role of a Supervisor agent in the supervisor agent pattern?

easy

A. To collect raw data from sensors

B. To train a single AI model

C. To replace all other agents with one

D. To manage and coordinate multiple AI agents

Solution

1. Step 1: Understand the supervisor agent's purpose

   The supervisor agent is designed to oversee and coordinate multiple AI agents working together.

2. Step 2: Differentiate from other roles

   Unlike training or data collection, the supervisor agent focuses on managing teamwork and quality control.

3. Final Answer:

   To manage and coordinate multiple AI agents -> Option D

4. Quick Check:

   Supervisor agent = manager of multiple agents [OK]

Hint: Supervisor agent = team manager of AI agents [OK]

Common Mistakes:

• Confusing supervisor with data collector
• Thinking supervisor trains models directly
• Assuming supervisor replaces all agents

2. Which of the following is the correct way to describe the supervisor agent's function in code?

easy

A. supervisor.replace_agents()

B. supervisor.train_single_model(data)

C. supervisor.collect_results(agents)

D. supervisor.ignore_agent_outputs()

Solution

1. Step 1: Identify supervisor's interaction with agents

   The supervisor collects and evaluates results from multiple agents, so a method like collect_results fits.

2. Step 2: Eliminate incorrect options

   Training a single model, replacing agents, or ignoring outputs do not match the supervisor's coordination role.

3. Final Answer:

   supervisor.collect_results(agents) -> Option C

4. Quick Check:

   Supervisor collects results = collect_results() [OK]

Hint: Supervisor collects and evaluates agent outputs [OK]

Common Mistakes:

• Choosing training method instead of collection
• Thinking supervisor replaces agents
• Ignoring outputs contradicts supervisor role

3. Given this code snippet for a supervisor agent pattern, what will be the printed output?

class Agent:
    def __init__(self, name, score):
        self.name = name
        self.score = score

class Supervisor:
    def __init__(self, agents):
        self.agents = agents
    def best_agent(self):
        return max(self.agents, key=lambda a: a.score).name

agents = [Agent('A1', 85), Agent('A2', 90), Agent('A3', 88)]
supervisor = Supervisor(agents)
print(supervisor.best_agent())

medium

A. A1

B. A2

C. A3

D. None

Solution

1. Step 1: Understand the agent scores

   Agents have scores: A1=85, A2=90, A3=88.

2. Step 2: Identify the agent with the highest score

   The best_agent method returns the name of the agent with the max score, which is A2 with 90.

3. Final Answer:

   A2 -> Option B

4. Quick Check:

   Max score agent = A2 [OK]

Hint: Max score agent name is printed [OK]

Common Mistakes:

• Choosing agent with second highest score
• Confusing agent names
• Assuming None if not found

4. Identify the bug in this supervisor agent code snippet:

class Supervisor:
    def __init__(self, agents):
        self.agents = agents
    def best_score(self):
        return max(self.agents, key=lambda a: a.score)

agents = [{'name': 'A1', 'score': 80}, {'name': 'A2', 'score': 95}]
supervisor = Supervisor(agents)
print(supervisor.best_score())

medium

A. Agents should be objects, not dictionaries

B. max() function is used incorrectly

C. Missing return statement in best_score

D. Supervisor class missing __init__ method

Solution

1. Step 1: Check agent data type and usage

   The best_score method expects agents with attribute score, but agents are dictionaries, not objects.

2. Step 2: Understand attribute vs key access

   Using a.score on a dictionary causes an error; dictionaries need a['score'].

3. Final Answer:

   Agents should be objects, not dictionaries -> Option A

4. Quick Check:

   Attribute access on dict causes error [OK]

Hint: Use objects or adjust attribute access for dicts [OK]

Common Mistakes:

• Thinking max() usage is wrong
• Missing return statement (it exists)
• Ignoring data type mismatch

5. You want to design a supervisor agent that combines outputs from three different AI agents solving a complex task. Which approach best fits the supervisor agent pattern?

hard

A. Collect outputs, evaluate quality, and select the best result

B. Run only the fastest agent and ignore others

C. Train all agents on the same data independently

D. Replace all agents with a single large model

Solution

1. Step 1: Understand supervisor agent's coordination role

   The supervisor should gather outputs from all agents and decide which is best based on quality.

2. Step 2: Evaluate other options

   Ignoring agents, training independently without coordination, or replacing agents contradict the supervisor pattern.

3. Final Answer:

   Collect outputs, evaluate quality, and select the best result -> Option A

4. Quick Check:

   Supervisor = collect + evaluate + select best [OK]

Hint: Supervisor picks best output from all agents [OK]

Common Mistakes:

• Ignoring some agents' outputs
• Confusing training with supervising
• Replacing agents instead of coordinating