Single agent vs multi-agent systems in Agentic AI - Metrics Comparison

Metrics & Evaluation - Single agent vs multi-agent systems

Which metric matters for this concept and WHY

In single agent systems, metrics like task success rate and efficiency matter because one agent tries to complete a goal alone. In multi-agent systems, coordination effectiveness, communication overhead, and collective reward are important to measure how well agents work together. These metrics help us understand if agents cooperate or compete successfully.

Confusion matrix or equivalent visualization (ASCII)

Single Agent Task Outcome:
+---------+-------+
| Success | Fail  |
+---------+-------+
|   80    |  20   |
+---------+-------+

Multi-Agent Coordination Outcome:
+-------------+-----------------+
| Coordinated | Not Coordinated |
+-------------+-----------------+
|     70      |       30        |
+-------------+-----------------+

Note: These simple tables show how many tasks succeeded alone or with coordination.
    

Precision vs Recall (or equivalent tradeoff) with concrete examples

In single agent systems, the tradeoff is often between speed and accuracy. For example, a robot vacuum might clean faster but miss spots (lower accuracy).

In multi-agent systems, the tradeoff is between coordination quality and communication cost. For example, many delivery drones working together can deliver faster (better coordination) but need more messages, which can slow them down or cause errors.

What "good" vs "bad" metric values look like for this use case

Single agent good: High task success rate (e.g., 95%), low time to complete task.

Single agent bad: Low success rate (e.g., 50%), long delays.

Multi-agent good: High coordination rate (e.g., 90%), low communication overhead, high collective reward.

Multi-agent bad: Poor coordination (e.g., 40%), high communication cost causing delays or conflicts.

Metrics pitfalls (accuracy paradox, data leakage, overfitting indicators)

• Ignoring coordination cost: Measuring only success without communication cost can hide inefficiencies in multi-agent systems.
• Overfitting to single tasks: Agents trained on one task may fail in new tasks, misleading success metrics.
• Data leakage: Sharing test data among agents can inflate performance falsely.
• Accuracy paradox: High success rate in simple tasks may not mean good performance in complex multi-agent scenarios.

Self-check: Your model has 98% accuracy but 12% recall on fraud. Is it good?

No, it is not good for fraud detection. The model misses many fraud cases (low recall), which is dangerous. High accuracy can be misleading if most data is non-fraud. For fraud, catching as many frauds as possible (high recall) is more important.

Key Result

Single agent systems focus on task success and efficiency, while multi-agent systems require balancing coordination quality and communication cost.

Practice

1. What is the main difference between a single agent system and a multi-agent system?

easy

A. A single agent system has one decision-maker, while a multi-agent system has multiple interacting agents.

B. A single agent system always uses deep learning, multi-agent systems do not.

C. Multi-agent systems cannot communicate, single agent systems can.

D. Single agent systems require more computing power than multi-agent systems.

Solution

1. Step 1: Understand agent count in systems

   Single agent systems have exactly one agent making decisions alone.

2. Step 2: Understand interaction in multi-agent systems

   Multi-agent systems have multiple agents that interact and cooperate or compete.

3. Final Answer:

   A single agent system has one decision-maker, while a multi-agent system has multiple interacting agents. -> Option A

4. Quick Check:

   Agent count and interaction define system type = A [OK]

Hint: Count agents: one means single, many means multi-agent [OK]

Common Mistakes:

• Confusing communication ability with agent count
• Thinking single agent systems always use deep learning
• Assuming multi-agent systems cannot communicate

2. Which of the following is the correct way to describe a multi-agent system?

easy

A. A system where one agent acts without any interaction.

B. A system that only uses a single neural network.

C. A system with multiple agents that can interact and collaborate.

D. A system that cannot learn from the environment.

Solution

1. Step 1: Identify multi-agent system traits

   Multi-agent systems have multiple agents that interact or collaborate.

2. Step 2: Eliminate incorrect options

   Descriptions of single agent without interaction, single neural network usage, or inability to learn are incorrect.

3. Final Answer:

   A system with multiple agents that can interact and collaborate. -> Option C

4. Quick Check:

   Multiple interacting agents = multi-agent system = C [OK]

Hint: Look for multiple interacting agents to spot multi-agent systems [OK]

Common Mistakes:

• Choosing single agent descriptions for multi-agent questions
• Confusing neural network use with agent count
• Assuming multi-agent systems cannot learn

3. Consider this Python code simulating agents' decisions:

class Agent:
    def __init__(self, name):
        self.name = name
    def act(self):
        return f"{self.name} acts alone"

agents = [Agent("A1"), Agent("A2")]
results = [agent.act() for agent in agents]
print(results)

What is the output?

medium

A. Error: Agent class missing act method

B. ['A1 acts alone']

C. ['acts alone', 'acts alone']

D. ['A1 acts alone', 'A2 acts alone']

Solution

1. Step 1: Understand the Agent class and act method

   Each Agent has a name and act() returns a string with that name plus 'acts alone'.

2. Step 2: List comprehension calls act() for each agent

   Two agents: 'A1' and 'A2', so results list has two strings with their names.

3. Final Answer:

   ['A1 acts alone', 'A2 acts alone'] -> Option D

4. Quick Check:

   Each agent acts alone string collected = A [OK]

Hint: List comprehension calls act() on each agent, so output matches agent names [OK]

Common Mistakes:

• Assuming only one agent acts
• Ignoring the agent name in the output string
• Thinking act method is missing

4. The following code is intended to simulate a multi-agent system where agents share their actions. What is the error?

class Agent:
    def __init__(self, name):
        self.name = name
    def act(self):
        return f"{self.name} acts"

agents = [Agent("A1"), Agent("A2")]
actions = []
for agent in agents:
    actions.append(agent.act)
print(actions)

medium

A. Agent class is missing the __init__ method.

B. The act method is not called; missing parentheses in append.

C. The list 'actions' should be a dictionary.

D. The print statement is outside the loop causing an error.

Solution

1. Step 1: Check how act method is used in the loop

   actions.append(agent.act) adds the method itself, not its result.

2. Step 2: Fix by calling the method with parentheses

   Use actions.append(agent.act()) to add the returned string.

3. Final Answer:

   The act method is not called; missing parentheses in append. -> Option B

4. Quick Check:

   Method call needs () to execute = B [OK]

Hint: Remember to call methods with () to get results, not the method itself [OK]

Common Mistakes:

• Appending method reference instead of calling it
• Thinking __init__ is missing when it is present
• Assuming print outside loop causes error

5. You want to design a system where multiple robots explore a building and share information to avoid collisions. Which system type fits best and why?

hard

A. Multi-agent system, because multiple robots interact and share information.

B. Multi-agent system, but agents act completely independently without sharing.

C. Single agent system, because robots do not need to communicate.

D. Single agent system, because one robot controls all decisions centrally.

Solution

1. Step 1: Analyze problem needs for multiple robots

   Multiple robots exploring means multiple agents acting simultaneously.

2. Step 2: Consider interaction and information sharing

   To avoid collisions, robots must share info and coordinate, needing interaction.

3. Final Answer:

   Multi-agent system, because multiple robots interact and share information. -> Option A

4. Quick Check:

   Multiple interacting agents sharing info = multi-agent system = D [OK]

Hint: Multiple robots sharing info means multi-agent system [OK]

Common Mistakes:

• Choosing single agent for multiple robots
• Ignoring need for communication to avoid collisions
• Thinking multi-agent means no interaction