Logging tool calls and results in Agentic AI - Model Metrics & Evaluation

Metrics & Evaluation - Logging tool calls and results

Which metric matters for Logging tool calls and results and WHY

When logging tool calls and results, the key metric is completeness and accuracy of logs. This means every tool call and its output should be recorded without missing or incorrect entries. This helps track what happened, when, and what the result was. It is important because it allows debugging, auditing, and understanding the system's behavior over time.

Confusion matrix or equivalent visualization

For logging, a confusion matrix is not directly applicable. Instead, a log completeness matrix can be imagined:

      +----------------------+---------------------+
      | Expected Logs        | Actual Logs         |
      +----------------------+---------------------+
      | Tool call made       | Tool call logged    |
      | Tool call made       | Tool call missing   |
      | Tool call not made   | No log entry        |
      +----------------------+---------------------+
    

We want all tool calls made to have matching log entries. Missing logs mean incomplete tracking.

Tradeoff: Completeness vs Performance

Logging every tool call and result can slow down the system (performance cost). If logs are too sparse, debugging becomes hard. If logs are too detailed, storage and speed suffer. The tradeoff is to log enough detail to understand behavior without overwhelming resources.

Example: Logging only errors is fast but misses successful calls. Logging all calls is thorough but slower.

What "good" vs "bad" logging looks like

Good logging: Every tool call is logged with timestamp, input, output, and status. Logs are easy to search and understand.

Bad logging: Missing logs for some calls, unclear or inconsistent format, no timestamps, or logs that do not show results.

Common pitfalls in logging tool calls and results

• Logging too little: Missing important calls or results.
• Logging too much: Huge logs that are hard to manage.
• Inconsistent formats: Hard to parse or analyze logs.
• Not logging errors or exceptions properly.
• Performance impact: Logging slows down the system if not optimized.

Self-check question

Your system logs 95% of tool calls but misses 5% randomly. Is this good? Why or why not?

Answer: This is not good because missing 5% of calls means some actions are not tracked. This can cause problems in debugging or auditing. Ideally, logging should be complete or near 100%.

Key Result

Complete and accurate logging of all tool calls and results is essential for reliable system tracking and debugging.

Practice

1. What is the main purpose of logging tool calls and results in DevOps?

easy

A. To make the tools run faster

B. To hide errors from users

C. To track what tools do and their outputs for debugging and monitoring

D. To reduce the size of log files

Solution

1. Step 1: Understand the role of logging

   Logging records actions and results of tools to help understand their behavior.

2. Step 2: Identify the benefits of logging

   Logging helps with debugging, monitoring, and auditing by showing what happened and when.

3. Final Answer:

   To track what tools do and their outputs for debugging and monitoring -> Option C

4. Quick Check:

   Logging = Track tool actions and outputs [OK]

Hint: Logging means recording tool actions and outputs clearly [OK]

Common Mistakes:

• Thinking logging speeds up tools
• Believing logging hides errors
• Assuming logging reduces log file size

2. Which of the following is the correct way to log a tool call and its result in a simple Python function?

easy

A. def log_call(tool_name, result): print(f"Tool {tool_name} result")

B. def log_call(tool_name, result): return f"Tool {tool_name} returned {result}"

C. def log_call(tool_name, result): print("Tool tool_name returned result")

D. def log_call(tool_name, result): print(f"Tool {tool_name} returned {result}")

Solution

1. Step 1: Check string formatting with variables

   def log_call(tool_name, result): print(f"Tool {tool_name} returned {result}") uses f-string correctly to insert variables tool_name and result.

2. Step 2: Verify output method

   def log_call(tool_name, result): print(f"Tool {tool_name} returned {result}") prints the message, which is typical for logging in simple scripts.

3. Final Answer:

   def log_call(tool_name, result): print(f"Tool {tool_name} returned {result}") -> Option D

4. Quick Check:

   Correct f-string and print used [OK]

Hint: Use f-strings and print() to log calls and results [OK]

Common Mistakes:

• Not using f-string for variable insertion
• Printing literal variable names instead of values
• Returning string instead of printing

3. Given the code below, what will be the output?

def log_call(tool, result):
    print(f"Calling {tool}...")
    print(f"Result: {result}")

log_call('BackupTool', 'Success')

medium

A. Calling BackupTool... Result: Success

B. Calling BackupTool...\nResult: Success

C. Calling tool...\nResult: result

D. Error: Missing parentheses in print

Solution

1. Step 1: Analyze the function calls

   The function prints two lines: one with tool name, one with result.

2. Step 2: Substitute arguments and check output

   Calling 'BackupTool' and 'Success' prints exactly two lines with those values.

3. Final Answer:

   Calling BackupTool...\nResult: Success -> Option B

4. Quick Check:

   Print lines match arguments [OK]

Hint: Read print lines carefully and substitute variables [OK]

Common Mistakes:

• Confusing variable names with strings
• Expecting output on one line
• Thinking print syntax is wrong

4. What is wrong with this logging function?

def log_call(tool, result):
    print("Calling tool...")
    print("Result: result")

medium

A. It prints the variable names instead of their values

B. It uses print instead of return

C. It has a syntax error in print statements

D. It logs too much information

Solution

1. Step 1: Check how variables are used in print

   The function prints literal strings "tool" and "result" instead of variable values.

2. Step 2: Understand correct variable usage

   To print values, variables must be inside f-strings or concatenated properly.

3. Final Answer:

   It prints the variable names instead of their values -> Option A

4. Quick Check:

   Variables not interpolated in strings [OK]

Hint: Use f-strings to print variable values, not names [OK]

Common Mistakes:

• Forgetting f before string
• Using quotes around variable names
• Thinking print must be replaced by return

5. You want to log multiple tool calls and their results in a list, showing each call and result clearly. Which code snippet correctly logs all calls from the list calls = [('ToolA', 'OK'), ('ToolB', 'Fail'), ('ToolC', 'OK')]?

hard

A. for tool, result in calls:
print(f"Tool {tool} returned {result}")

B. for call in calls:
print(f"Tool call[0] returned call[1]")

C. for tool, result in calls:
print("Tool tool returned result")

D. for tool, result in calls:
print(f"Tool {tool} result")

Solution

1. Step 1: Understand tuple unpacking in loop

   for tool, result in calls: print(f"Tool {tool} returned {result}") correctly unpacks each tuple into tool and result variables.

2. Step 2: Check correct f-string usage

   for tool, result in calls: print(f"Tool {tool} returned {result}") uses f-string to insert variables properly in the print statement.

3. Final Answer:

   for tool, result in calls: print(f"Tool {tool} returned {result}") -> Option A

4. Quick Check:

   Tuple unpacking and f-string correct [OK]

Hint: Unpack tuples and use f-strings to log each call [OK]

Common Mistakes:

• Not unpacking tuples correctly
• Printing variable names as strings
• Missing f-string for variable insertion