Parallel tool execution in Agentic AI - Model Metrics & Evaluation

Metrics & Evaluation - Parallel tool execution

Which metric matters for Parallel tool execution and WHY

When running multiple tools or models at the same time, the key metric is throughput. Throughput measures how many tasks or requests are completed in a given time. It shows how well the system handles parallel work.

Another important metric is latency, which is the time taken to complete a single task. Low latency means faster responses.

We also watch resource utilization to ensure the system uses CPU, memory, and other resources efficiently without overload.

Confusion matrix or equivalent visualization

Parallel tool execution does not use a confusion matrix like classification tasks. Instead, we use performance charts such as:

Throughput over time:
| Time (s) | Tasks Completed |
|----------|-----------------|
| 1        | 100             |
| 2        | 210             |
| 3        | 320             |

Latency distribution:
| Latency (ms) | Count |
|--------------|-------|
| 10-20        | 150   |
| 20-30        | 80    |
| 30-40        | 20    |
    

These help us see how many tasks finish quickly and how many take longer.

Precision vs Recall tradeoff analogy for Parallel tool execution

In parallel execution, the tradeoff is between maximizing throughput and minimizing latency.

If we push for very high throughput by running many tools at once, latency might increase because resources get crowded.

If we focus on low latency by limiting parallel tasks, throughput might drop because fewer tasks run at the same time.

Example: A web server handling many requests simultaneously (high throughput) might slow down individual responses (higher latency). Balancing these is key.

What "good" vs "bad" metric values look like for Parallel tool execution

Good:

• High throughput: many tasks completed per second.
• Low latency: most tasks finish quickly.
• Balanced resource use: CPU and memory are well used but not overloaded.

Bad:

• Low throughput: few tasks done over time.
• High latency: tasks take too long to finish.
• Resource overload: CPU or memory maxed out causing slowdowns or crashes.

Metrics pitfalls in Parallel tool execution

• Ignoring latency: Focusing only on throughput can hide slow responses.
• Resource bottlenecks: Not monitoring CPU or memory can cause crashes.
• Uneven load: Some tools may run slower, causing delays.
• Overfitting to test data: Optimizing metrics only on small tests may not work in real use.

Self-check question

Your parallel system completes 1000 tasks per second (high throughput) but some tasks take 10 seconds to finish (high latency). Is this good?

Answer: Not fully. While throughput is good, high latency means some tasks are very slow. This can hurt user experience or downstream processes. You should balance throughput and latency better.

Key Result

Throughput and latency are key metrics to balance for efficient parallel tool execution.

Practice

1.

What is the main benefit of parallel tool execution in AI workflows?

easy

A. It makes tools run slower but more accurately.

B. It runs tools one after another to avoid errors.

C. It only works if tasks depend on each other.

D. It runs multiple tools at the same time to save time.

Solution

1. Step 1: Understand parallel execution

   Parallel execution means running many tasks at once, not one by one.

2. Step 2: Identify the benefit in AI workflows

   Running tools simultaneously saves time and improves efficiency.

3. Final Answer:

   It runs multiple tools at the same time to save time. -> Option D

4. Quick Check:

   Parallel execution = run many tools at once [OK]

Hint: Parallel means many at once, so it saves time [OK]

Common Mistakes:

• Thinking parallel means slower execution
• Believing tasks must depend on each other
• Confusing parallel with sequential execution

2.

Which of the following is the correct way to start parallel execution of two tools toolA and toolB in Python using concurrent.futures?

import concurrent.futures

with concurrent.futures.ThreadPoolExecutor() as executor:
    # What goes here?

easy

A. executor.parallel(toolA, toolB)

B. executor.run(toolA, toolB)

C. executor.submit(toolA); executor.submit(toolB)

D. executor.execute(toolA & toolB)

Solution

1. Step 1: Recall ThreadPoolExecutor usage

   The method to run functions in parallel is submit() for each function.

2. Step 2: Check the options

   Only executor.submit(toolA); executor.submit(toolB) correctly submits both tools for parallel execution.

3. Final Answer:

   executor.submit(toolA); executor.submit(toolB) -> Option C

4. Quick Check:

   Use submit() to run functions in parallel [OK]

Hint: Use submit() for each tool to run in parallel [OK]

Common Mistakes:

• Using non-existent methods like run() or execute()
• Trying to pass multiple tools in one call
• Confusing parallel execution syntax

3.

Given the code below, what will be the output?

import concurrent.futures
import time

def tool1():
    time.sleep(2)
    return 'Done1'

def tool2():
    time.sleep(1)
    return 'Done2'

with concurrent.futures.ThreadPoolExecutor() as executor:
    future1 = executor.submit(tool1)
    future2 = executor.submit(tool2)
    print(future1.result())
    print(future2.result())

medium

A. Done2\nDone1

B. Done1\nDone2

C. Done1\nDone1

D. Done2\nDone2

Solution

1. Step 1: Understand parallel execution and sleep times

   tool1 sleeps 2 seconds, tool2 sleeps 1 second, both start together.

2. Step 2: Check order of result() calls

   future1.result() waits for tool1 (2s), then future2.result() waits for tool2 (1s). So output order matches calls, not completion time.

3. Final Answer:

   Done1 Done2 -> Option B

4. Quick Check:

   Results print in call order, not finish order [OK]

Hint: result() waits; output order matches calls, not finish time [OK]

Common Mistakes:

• Assuming output order matches task finish time
• Ignoring that result() blocks until done
• Confusing sleep durations with print order

4.

What is the error in the following code that tries to run two tools in parallel?

import concurrent.futures

def toolA():
    return 'A'

def toolB():
    return 'B'

with concurrent.futures.ThreadPoolExecutor() as executor:
    results = executor.map(toolA, toolB)
    print(list(results))

medium

A. executor.map expects a function and an iterable, but toolB is not iterable.

B. executor.map cannot run more than one function at a time.

C. Missing parentheses when calling toolA and toolB.

D. ThreadPoolExecutor cannot be used with map.

Solution

1. Step 1: Understand executor.map signature

   executor.map expects one function and an iterable of inputs to apply that function to.

2. Step 2: Identify the error in arguments

   Passing two functions (toolA, toolB) is wrong; toolB is not an iterable of inputs for toolA.

3. Final Answer:

   executor.map expects a function and an iterable, but toolB is not iterable. -> Option A

4. Quick Check:

   map(func, iterable) needs iterable inputs [OK]

Hint: map needs one function and iterable inputs, not two functions [OK]

Common Mistakes:

• Passing multiple functions to map
• Confusing map with submit
• Thinking map runs multiple different functions

5.

You want to run three independent AI tools toolX, toolY, and toolZ in parallel and collect their results as a dictionary with tool names as keys. Which code snippet correctly achieves this?

def toolX(): return 'X result'
def toolY(): return 'Y result'
def toolZ(): return 'Z result'

# Choose the correct parallel execution code

hard

A. import concurrent.futures
with concurrent.futures.ThreadPoolExecutor() as executor:
futures = {name: executor.submit(func) for name, func in {'toolX': toolX, 'toolY': toolY, 'toolZ': toolZ}.items()}
results = {name: future.result() for name, future in futures.items()}
print(results)

B. import concurrent.futures
with concurrent.futures.ThreadPoolExecutor() as executor:
results = executor.map(toolX, toolY, toolZ)
print(dict(results))

C. results = {}
for tool in [toolX, toolY, toolZ]:
results[tool.__name__] = tool()
print(results)

D. import threading
results = {}
def run_tool(name, func):
results[name] = func()
threads = []
for name, func in {'toolX': toolX, 'toolY': toolY, 'toolZ': toolZ}.items():
t = threading.Thread(target=run_tool, args=(name, func))
threads.append(t)
t.start()
for t in threads:
t.join()
print(results)

Solution

1. Step 1: Check parallel execution with ThreadPoolExecutor

   import concurrent.futures with concurrent.futures.ThreadPoolExecutor() as executor: futures = {name: executor.submit(func) for name, func in {'toolX': toolX, 'toolY': toolY, 'toolZ': toolZ}.items()} results = {name: future.result() for name, future in futures.items()} print(results) uses submit() for each tool, stores futures with names, then collects results correctly.

2. Step 2: Evaluate other options

   import concurrent.futures with concurrent.futures.ThreadPoolExecutor() as executor: results = executor.map(toolX, toolY, toolZ) print(dict(results)) misuses map with multiple functions; results = {} for tool in [toolX, toolY, toolZ]: results[tool.__name__] = tool() print(results) runs tools sequentially; import threading results = {} def run_tool(name, func): results[name] = func() threads = [] for name, func in {'toolX': toolX, 'toolY': toolY, 'toolZ': toolZ}.items(): t = threading.Thread(target=run_tool, args=(name, func)) threads.append(t) t.start() for t in threads: t.join() print(results) uses threading correctly but is more complex and not asked.

3. Final Answer:

   Option A correctly runs tools in parallel and collects results as a dictionary. -> Option A

4. Quick Check:

   submit() + dict comprehension collects parallel results [OK]

Hint: Use submit() with dict comprehension to map names to results [OK]

Common Mistakes:

• Using map() with multiple functions
• Running tools sequentially instead of parallel
• Confusing threading with ThreadPoolExecutor usage