Handling tool execution results in Agentic AI - Model Metrics & Evaluation

Metrics & Evaluation - Handling tool execution results

Which metric matters for Handling tool execution results and WHY

When handling tool execution results in AI agents, the key metrics are accuracy and execution success rate. Accuracy tells us how often the tool's output matches the expected result. Execution success rate shows how often the tool runs without errors. These metrics matter because a tool that runs correctly but gives wrong results is not useful, and a tool that often fails to run is unreliable. Together, they help us trust the AI agent's decisions based on tool outputs.

Confusion matrix for tool execution results

Imagine a tool that classifies tasks as successful or failed. The confusion matrix looks like this:

      | Predicted Success | Predicted Failure |
      |-------------------|-------------------|
      | True Success (TP)  | False Failure (FN) |
      | False Success (FP) | True Failure (TN)  |
    

Where:

• TP: Tool executed and result was correct.
• FP: Tool said success but result was wrong.
• FN: Tool said failure but result was actually correct.
• TN: Tool failed and result was wrong.

From this, we calculate precision and recall to understand tool reliability.

Precision vs Recall tradeoff with examples

Precision measures how often the tool's successful execution results are actually correct. High precision means few wrong successes.

Recall measures how many of the actual correct results the tool successfully executes. High recall means the tool rarely misses correct executions.

Example:

• If the AI agent controls a medical tool, high recall is critical to not miss any correct diagnoses.
• If the AI agent controls a financial transaction tool, high precision is important to avoid false successful transactions.

Balancing precision and recall depends on the tool's purpose and risk tolerance.

What good vs bad metric values look like for Handling tool execution results

Good metrics:

• Accuracy above 90% means the tool usually gives correct results.
• Precision and recall both above 85% show balanced and reliable execution.
• Low failure rate (less than 5%) means the tool rarely crashes or errors.

Bad metrics:

• Accuracy below 70% means many wrong results.
• Precision very low (e.g., 50%) means many false successful executions.
• Recall very low (e.g., 40%) means many missed correct executions.
• High failure rate (above 20%) means the tool is unstable.

Common pitfalls in metrics for Handling tool execution results

• Accuracy paradox: If most executions are failures, a tool that always fails can show high accuracy but is useless.
• Data leakage: Using future or test data to evaluate tool results can give overly optimistic metrics.
• Overfitting: Tool tuned too much on training data may fail on new tasks, showing poor real-world metrics.
• Ignoring failure types: Not distinguishing between execution errors and wrong results can hide issues.

Self-check question

Your AI agent's tool has 98% accuracy but only 12% recall on successful executions. Is this good for production?

Answer: No. The tool rarely catches correct executions (low recall), so it misses many good results. Despite high accuracy, it is not reliable for production because it fails to perform its task well.

Key Result

For handling tool execution results, balanced high precision and recall ensure reliable and correct tool outputs.

Practice

1. What is the main reason an AI agent should carefully handle the results returned by a tool it uses?

easy

A. To reduce the size of the tool's code

B. To make the tool run faster

C. To ensure the agent makes correct decisions based on accurate information

D. To avoid using any external resources

Solution

1. Step 1: Understand the role of tool results in AI agents

   AI agents rely on tools to get extra information or perform tasks that help them decide what to do next.

2. Step 2: Recognize the importance of accurate results

   If the agent does not handle the tool's results carefully, it might make wrong decisions based on incorrect or incomplete data.

3. Final Answer:

   To ensure the agent makes correct decisions based on accurate information -> Option C

4. Quick Check:

   Handling results carefully = correct decisions [OK]

Hint: Focus on why accuracy matters for agent decisions [OK]

Common Mistakes:

• Thinking speed of tool matters more than result accuracy
• Ignoring the importance of result correctness
• Confusing tool code size with result handling

2. Which of the following is the correct way to check if a tool's execution result is empty in Python before using it?

easy

A. if result is None:

B. if result != None:

C. if result = None:

D. if result == None:

Solution

1. Step 1: Identify the correct syntax for None comparison in Python

   In Python, to check if a variable is None, use 'is None' instead of '==' because None is a singleton.

2. Step 2: Eliminate incorrect options

   if result == None: uses '==', which works but is not recommended. if result = None: uses '=' which is assignment, causing syntax error. if result != None: checks for not None, which is opposite.

3. Final Answer:

   if result is None: -> Option A

4. Quick Check:

   Use 'is None' to check None in Python [OK]

Hint: Use 'is None' to check for None, not '==' or '=' [OK]

Common Mistakes:

• Using '=' instead of '==' or 'is' causing syntax errors
• Using '==' instead of 'is' for None comparison
• Checking for not None when expecting None

3. Given the code below, what will be printed?

tool_result = {'status': 'success', 'data': [1, 2, 3]}
if tool_result.get('status') == 'success':
    print(len(tool_result['data']))
else:
    print(0)

medium

A. KeyError

B. 0

C. None

D. 3

Solution

1. Step 1: Check the status key in tool_result

   tool_result.get('status') returns 'success', so the if condition is True.

2. Step 2: Calculate length of data list

   tool_result['data'] is [1, 2, 3], which has length 3, so print(3) is executed.

3. Final Answer:

   3 -> Option D

4. Quick Check:

   Status is 'success', print length 3 [OK]

Hint: Check condition first, then count list length [OK]

Common Mistakes:

• Assuming else branch runs
• Confusing get() with direct key access
• Expecting KeyError when key exists

4. What is the error in the following code snippet that handles a tool's result?

result = tool.run()
if result != None:
    print(result['value'])
else:
    print('No result')

medium

A. Using '!=' instead of 'is not' to check None

B. Missing try-except block for key access

C. Using print instead of return

D. No error, code is correct

Solution

1. Step 1: Analyze None check

   Using 'result != None' works but 'result is not None' is preferred; this is not a critical error.

2. Step 2: Check key access safety

   Accessing result['value'] without checking if 'value' exists can cause KeyError if missing; no try-except or key check is present.

3. Final Answer:

   Missing try-except block for key access -> Option B

4. Quick Check:

   Always handle missing keys safely [OK]

Hint: Always check keys or catch exceptions when accessing dict values [OK]

Common Mistakes:

• Ignoring possible missing keys causing runtime errors
• Thinking '!=' None is always wrong
• Confusing print and return usage

5. An AI agent uses a tool that returns a dictionary with keys 'status' and 'output'. Sometimes 'output' can be an empty string or None. Which is the best way to handle the tool's result to safely get meaningful output or fallback to 'No data'?

hard

A. if result.get('status') == 'success' and result.get('output'):
use_output = result['output']
else:
use_output = 'No data'

B. if result['status'] == 'success' and result['output'] != '':
use_output = result['output']
else:
use_output = 'No data'

C. if result.get('status') == 'success' and result['output'] is not None:
use_output = result['output']
else:
use_output = 'No data'

D. if result['status'] == 'success' and result['output']:
use_output = result['output']
else:
use_output = 'No data'

Solution

1. Step 1: Use safe key access with get()

   Using result.get('status') avoids KeyError if 'status' is missing, making code safer.

2. Step 2: Check output truthiness to handle empty string or None

   Checking 'and result.get('output')' ensures output is not None or empty string, both falsy values, so fallback triggers correctly.

3. Final Answer:

   Option A -> Option A

4. Quick Check:

   Safe get() and truthy check handle missing or empty output [OK]

Hint: Use get() and check truthiness for safe, clean handling [OK]

Common Mistakes:

• Using direct key access risking KeyError
• Checking only for None but missing empty string case
• Not handling missing keys safely