Async agent execution in Agentic AI - Model Metrics & Evaluation

Metrics & Evaluation - Async agent execution

Which metric matters for Async agent execution and WHY

When running agents asynchronously, key metrics include throughput (how many tasks finish per time), latency (time to complete each task), and success rate (how many tasks finish correctly). These show if the system is fast, responsive, and reliable. Accuracy of the agent's output is also important to measure quality.

Confusion matrix or equivalent visualization

Async Agent Task Results:

| Task ID | Status   | Result Correct? |
|---------|----------|-----------------|
| 1       | Success  | Yes             |
| 2       | Success  | No              |
| 3       | Failed   | N/A             |
| 4       | Success  | Yes             |
| 5       | Success  | Yes             |

Summary:
- Total tasks: 5
- Success: 4
- Failures: 1
- Correct results: 3

Metrics:
- Success Rate = 4/5 = 0.8
- Accuracy (on success) = 3/4 = 0.75
- Overall Accuracy = 3/5 = 0.6
    

Precision vs Recall tradeoff with concrete examples

In async agent execution, precision means how many completed tasks are actually correct. Recall means how many correct tasks the system completes out of all tasks that should be done.

Example: If the agent completes many tasks quickly but some are wrong, precision is low. If it completes only a few tasks but all are correct, recall is low.

Choosing between speed and correctness depends on use case. For urgent tasks, higher recall (completing more tasks) may be better. For critical tasks, higher precision (correct results) matters more.

What "good" vs "bad" metric values look like for async agent execution

Good: Success rate above 90%, accuracy above 85%, low latency (tasks finish quickly), and high throughput (many tasks done per second).

Bad: Success rate below 70%, accuracy below 60%, high latency (slow task completion), and low throughput (few tasks done).

Good metrics mean the async agent is fast, reliable, and produces correct results. Bad metrics mean delays, failures, or wrong outputs.

Common pitfalls in metrics for async agent execution

• Ignoring failed tasks: Only measuring successful tasks can hide failure problems.
• Data leakage: Using future info to evaluate current tasks inflates accuracy.
• Overfitting: Agent may perform well on test tasks but fail on new ones.
• Latency spikes: Average latency hides occasional very slow tasks.
• Throughput vs quality tradeoff: Maximizing speed may reduce accuracy.

Self-check question

Your async agent has 98% success rate but only 12% recall on critical tasks. Is it good for production? Why or why not?

Answer: No, it is not good. Although most tasks finish successfully, the agent misses many critical tasks (low recall). This means important work is not done, which can cause serious problems.

Key Result

For async agent execution, balance success rate, accuracy, latency, and throughput to ensure fast, reliable, and correct task completion.

Practice

1. What is the main benefit of using async agent execution in AI systems?

easy

A. It makes the agents run slower but more accurately.

B. It allows multiple agents to run at the same time, speeding up processing.

C. It forces agents to run one after another in a fixed order.

D. It disables agents from communicating with each other.

Solution

1. Step 1: Understand async execution

   Async execution means running tasks without waiting for each to finish before starting the next.

2. Step 2: Apply to AI agents

   Running multiple AI agents at the same time speeds up overall processing by avoiding delays.

3. Final Answer:

   It allows multiple agents to run at the same time, speeding up processing. -> Option B

4. Quick Check:

   Async = concurrent execution = speed up [OK]

Hint: Async means agents run together, not one by one [OK]

Common Mistakes:

• Thinking async slows down agents
• Believing async forces sequential runs
• Confusing async with disabling communication

2. Which of the following is the correct syntax to run multiple async agents together in Python?

easy

A. await agent1() and agent2()

B. asyncio.run(agent1(), agent2())

C. await asyncio.gather(agent1(), agent2())

D. async gather(agent1, agent2)

Solution

1. Step 1: Recall asyncio syntax

   To run multiple async functions concurrently, use await asyncio.gather(...).

2. Step 2: Check options

   await asyncio.gather(agent1(), agent2()) uses correct syntax with await asyncio.gather(agent1(), agent2()). Others are invalid or incorrect.

3. Final Answer:

   await asyncio.gather(agent1(), agent2()) -> Option C

4. Quick Check:

   asyncio.gather + await = correct syntax [OK]

Hint: Use await with asyncio.gather to run agents together [OK]

Common Mistakes:

• Using asyncio.run with multiple args
• Missing await before asyncio.gather
• Wrong function call syntax without parentheses

3. Given the code below, what will be the output?

import asyncio

async def agent1():
    await asyncio.sleep(1)
    return 'Agent1 done'

async def agent2():
    await asyncio.sleep(2)
    return 'Agent2 done'

async def main():
    results = await asyncio.gather(agent1(), agent2())
    print(results)

asyncio.run(main())

medium

A. ['Agent1 done', 'Agent2 done'] after about 2 seconds

B. ['Agent2 done', 'Agent1 done'] after about 2 seconds

C. ['Agent1 done', 'Agent2 done'] after about 3 seconds

D. Error because agent2 takes longer

Solution

1. Step 1: Understand asyncio.gather timing

   asyncio.gather runs tasks concurrently, so total time is max of individual times.

2. Step 2: Analyze sleep durations

   agent1 sleeps 1s, agent2 sleeps 2s, so total time ~2 seconds, results in order of calls.

3. Final Answer:

   ['Agent1 done', 'Agent2 done'] after about 2 seconds -> Option A

4. Quick Check:

   Concurrent run time = max sleep = 2s [OK]

Hint: Total time = longest agent sleep with asyncio.gather [OK]

Common Mistakes:

• Adding sleep times instead of taking max
• Assuming output order changes by sleep time
• Expecting error due to different sleep durations

4. What is wrong with this async agent execution code?

import asyncio

async def agent():
    return 'done'

async def main():
    results = asyncio.gather(agent(), agent())
    print(results)

asyncio.run(main())

medium

A. Missing await before asyncio.gather, so results is a coroutine, not actual results.

B. agent() is not async, so cannot be awaited.

C. asyncio.run cannot be used with async functions.

D. print cannot be used inside async functions.

Solution

1. Step 1: Check asyncio.gather usage

   asyncio.gather returns a coroutine; it must be awaited to get results.

2. Step 2: Identify missing await

   Code misses await before asyncio.gather, so print shows coroutine object, not results.

3. Final Answer:

   Missing await before asyncio.gather, so results is a coroutine, not actual results. -> Option A

4. Quick Check:

   Always await asyncio.gather to get results [OK]

Hint: Always put await before asyncio.gather to get results [OK]

Common Mistakes:

• Forgetting await before asyncio.gather
• Thinking print can't be used in async
• Misunderstanding asyncio.run usage

5. You want to run three async agents where agent3 depends on the results of agent1 and agent2. Which approach correctly handles this dependency using async agent execution?

hard

A. Run all three agents sequentially without async to ensure order.

B. Run agent3 concurrently with agent1 and agent2 using asyncio.gather without waiting.

C. Run agent3 first, then run agent1 and agent2 concurrently after.

D. Run agent1 and agent2 concurrently with asyncio.gather, await their results, then run agent3 with those results.

Solution

1. Step 1: Identify dependency order

   agent3 needs results from agent1 and agent2, so it must run after they finish.

2. Step 2: Use asyncio.gather for parallelism

   Run agent1 and agent2 concurrently with asyncio.gather, await results, then pass to agent3.

3. Final Answer:

   Run agent1 and agent2 concurrently with asyncio.gather, await their results, then run agent3 with those results. -> Option D

4. Quick Check:

   Run dependencies first, then dependent agent [OK]

Hint: Await dependencies before running dependent agent [OK]

Common Mistakes:

• Running dependent agent before dependencies finish
• Running all agents concurrently ignoring dependencies
• Running sequentially losing async speed benefits