Logging tool calls and results in Agentic AI - Model Pipeline Trace

Model Pipeline - Logging tool calls and results

This pipeline shows how an AI agent logs each tool call it makes and records the results. It helps track what the agent did and what answers it got back.

Data Flow - 5 Stages

1Input Query

1 query string→Receive user question or command→1 query string

"What is the weather today?"

↓

2Tool Call Preparation

1 query string→Decide which tool to call and prepare parameters→1 tool call object

{"tool": "weather_api", "params": {"location": "New York"}}

↓

3Tool Call Execution

1 tool call object→Call the external tool and get result→1 tool result object

{"temperature": "22C", "condition": "Sunny"}

↓

4Logging Tool Call

1 tool call object + 1 tool result object→Save the tool call and its result to log→1 log entry

{"tool": "weather_api", "params": {"location": "New York"}, "result": {"temperature": "22C", "condition": "Sunny"}, "timestamp": "2024-06-01T10:00:00Z"}

↓

5Response Generation

1 tool result object→Create a user-friendly answer from tool result→1 response string

"The weather in New York is 22 degrees Celsius and sunny."

Training Trace - Epoch by Epoch

Loss
0.5 |****
0.4 |****
0.3 |***
0.2 |**
0.1 |*
0.0 +----------------
      1 2 3 4 5 Epochs

Epoch	Loss ↓	Accuracy ↑	Observation
1	0.45	0.60	Initial training with random tool call logs, model starts learning to predict correct tool usage.
2	0.30	0.75	Model improves in selecting correct tools and logging results accurately.
3	0.20	0.85	Better tool call predictions and consistent logging behavior.
4	0.15	0.90	Model converges with reliable tool call and logging accuracy.
5	0.12	0.92	Final fine-tuning, stable and accurate logging of tool calls and results.

Prediction Trace - 5 Layers

Layer 1: Receive Query

Layer 2: Select Tool and Prepare Call

Layer 3: Execute Tool Call

Layer 4: Log Tool Call and Result

Layer 5: Generate Response

Model Quiz - 3 Questions

What is the main purpose of logging tool calls and results?

A To keep track of what tools were used and their outputs

B To speed up the tool calls

C To change the tool outputs

D To delete old tool calls

Key Insight

Logging tool calls and results helps the AI agent keep a clear record of its actions and outputs. This makes it easier to debug, improve, and trust the agent's behavior over time.

Practice

1. What is the main purpose of logging tool calls and results in DevOps?

easy

A. To make the tools run faster

B. To hide errors from users

C. To track what tools do and their outputs for debugging and monitoring

D. To reduce the size of log files

Solution

1. Step 1: Understand the role of logging

   Logging records actions and results of tools to help understand their behavior.

2. Step 2: Identify the benefits of logging

   Logging helps with debugging, monitoring, and auditing by showing what happened and when.

3. Final Answer:

   To track what tools do and their outputs for debugging and monitoring -> Option C

4. Quick Check:

   Logging = Track tool actions and outputs [OK]

Hint: Logging means recording tool actions and outputs clearly [OK]

Common Mistakes:

• Thinking logging speeds up tools
• Believing logging hides errors
• Assuming logging reduces log file size

2. Which of the following is the correct way to log a tool call and its result in a simple Python function?

easy

A. def log_call(tool_name, result): print(f"Tool {tool_name} result")

B. def log_call(tool_name, result): return f"Tool {tool_name} returned {result}"

C. def log_call(tool_name, result): print("Tool tool_name returned result")

D. def log_call(tool_name, result): print(f"Tool {tool_name} returned {result}")

Solution

1. Step 1: Check string formatting with variables

   def log_call(tool_name, result): print(f"Tool {tool_name} returned {result}") uses f-string correctly to insert variables tool_name and result.

2. Step 2: Verify output method

   def log_call(tool_name, result): print(f"Tool {tool_name} returned {result}") prints the message, which is typical for logging in simple scripts.

3. Final Answer:

   def log_call(tool_name, result): print(f"Tool {tool_name} returned {result}") -> Option D

4. Quick Check:

   Correct f-string and print used [OK]

Hint: Use f-strings and print() to log calls and results [OK]

Common Mistakes:

• Not using f-string for variable insertion
• Printing literal variable names instead of values
• Returning string instead of printing

3. Given the code below, what will be the output?

def log_call(tool, result):
    print(f"Calling {tool}...")
    print(f"Result: {result}")

log_call('BackupTool', 'Success')

medium

A. Calling BackupTool... Result: Success

B. Calling BackupTool...\nResult: Success

C. Calling tool...\nResult: result

D. Error: Missing parentheses in print

Solution

1. Step 1: Analyze the function calls

   The function prints two lines: one with tool name, one with result.

2. Step 2: Substitute arguments and check output

   Calling 'BackupTool' and 'Success' prints exactly two lines with those values.

3. Final Answer:

   Calling BackupTool...\nResult: Success -> Option B

4. Quick Check:

   Print lines match arguments [OK]

Hint: Read print lines carefully and substitute variables [OK]

Common Mistakes:

• Confusing variable names with strings
• Expecting output on one line
• Thinking print syntax is wrong

4. What is wrong with this logging function?

def log_call(tool, result):
    print("Calling tool...")
    print("Result: result")

medium

A. It prints the variable names instead of their values

B. It uses print instead of return

C. It has a syntax error in print statements

D. It logs too much information

Solution

1. Step 1: Check how variables are used in print

   The function prints literal strings "tool" and "result" instead of variable values.

2. Step 2: Understand correct variable usage

   To print values, variables must be inside f-strings or concatenated properly.

3. Final Answer:

   It prints the variable names instead of their values -> Option A

4. Quick Check:

   Variables not interpolated in strings [OK]

Hint: Use f-strings to print variable values, not names [OK]

Common Mistakes:

• Forgetting f before string
• Using quotes around variable names
• Thinking print must be replaced by return

5. You want to log multiple tool calls and their results in a list, showing each call and result clearly. Which code snippet correctly logs all calls from the list calls = [('ToolA', 'OK'), ('ToolB', 'Fail'), ('ToolC', 'OK')]?

hard

A. for tool, result in calls:
print(f"Tool {tool} returned {result}")

B. for call in calls:
print(f"Tool call[0] returned call[1]")

C. for tool, result in calls:
print("Tool tool returned result")

D. for tool, result in calls:
print(f"Tool {tool} result")

Solution

1. Step 1: Understand tuple unpacking in loop

   for tool, result in calls: print(f"Tool {tool} returned {result}") correctly unpacks each tuple into tool and result variables.

2. Step 2: Check correct f-string usage

   for tool, result in calls: print(f"Tool {tool} returned {result}") uses f-string to insert variables properly in the print statement.

3. Final Answer:

   for tool, result in calls: print(f"Tool {tool} returned {result}") -> Option A

4. Quick Check:

   Tuple unpacking and f-string correct [OK]

Hint: Unpack tuples and use f-strings to log each call [OK]

Common Mistakes:

• Not unpacking tuples correctly
• Printing variable names as strings
• Missing f-string for variable insertion