Supervisor agent pattern in Agentic AI - Model Metrics & Evaluation

Metrics & Evaluation - Supervisor agent pattern

Which metric matters for the Supervisor agent pattern and WHY

The Supervisor agent pattern involves a main agent overseeing other agents to ensure correct task completion. The key metric here is accuracy of the supervisor's decisions to accept or reject sub-agent outputs. This accuracy ensures the supervisor correctly identifies good or bad outputs, maintaining overall system reliability.

Additionally, precision and recall are important to measure how well the supervisor balances catching errors (recall) without wrongly rejecting good outputs (precision).

Confusion matrix for Supervisor agent decisions

      | Predicted Good       | Predicted Bad        |
      |---------------------|----------------------|
      | True Positive  (TP)  | False Negative (FN)  |
      | False Positive (FP)  | True Negative  (TN)  |

      TP: Supervisor correctly accepts good output
      FP: Supervisor wrongly accepts bad output
      FN: Supervisor wrongly rejects good output
      TN: Supervisor correctly rejects bad output
    

Precision vs Recall tradeoff in Supervisor agent pattern

Precision measures how many accepted outputs are truly good. High precision means the supervisor rarely accepts bad outputs, avoiding errors downstream.

Recall measures how many good outputs the supervisor correctly accepts. High recall means the supervisor rarely rejects good outputs, avoiding unnecessary rework.

Tradeoff example: If the supervisor is too strict, recall drops (many good outputs rejected), causing delays. If too lenient, precision drops (bad outputs accepted), causing errors.

What "good" vs "bad" metric values look like for Supervisor agent pattern

• Good: Accuracy > 90%, Precision > 85%, Recall > 85% - Supervisor reliably accepts good outputs and rejects bad ones.
• Bad: Accuracy < 70%, Precision < 60%, Recall < 60% - Supervisor often makes wrong decisions, harming system trust.

Common pitfalls in evaluating Supervisor agent pattern metrics

• Accuracy paradox: If bad outputs are rare, high accuracy can be misleading if supervisor always accepts outputs.
• Data leakage: Using future or test data to train supervisor inflates metrics falsely.
• Overfitting: Supervisor tuned too closely to training data may fail on new outputs.

Self-check question

Your supervisor agent has 98% accuracy but only 12% recall on bad outputs. Is it good for production? Why or why not?

Answer: No, it is not good. The supervisor misses 88% of bad outputs (low recall), allowing many errors through. High accuracy is misleading because bad outputs are rare. Improving recall is critical to catch errors.

Key Result

Supervisor agent pattern needs balanced precision and recall to reliably accept good outputs and reject bad ones.