Defining tool schemas and descriptions in Agentic AI - Model Metrics & Evaluation

Metrics & Evaluation - Defining tool schemas and descriptions

Which metric matters for Defining tool schemas and descriptions and WHY

When defining tool schemas and descriptions, the key metric is accuracy of the schema in representing the tool's capabilities and constraints. This ensures the tool behaves as expected when used by AI agents. Accuracy here means the schema correctly captures all required inputs, outputs, and usage rules without errors.

Why accuracy? Because an incorrect schema can cause the AI to misuse the tool, leading to wrong results or failures. Clear, precise descriptions help the AI understand how to call the tool properly.

Confusion matrix or equivalent visualization

Schema Validation Confusion Matrix (Example):

               | Correct Schema | Incorrect Schema
---------------|----------------|-----------------
Predicted Good |       90       |        5        
Predicted Bad  |       3        |       12        

- True Positives (TP): 90 (correctly identified good schemas)
- False Positives (FP): 5 (incorrectly accepted bad schemas)
- False Negatives (FN): 3 (missed good schemas)
- True Negatives (TN): 12 (correctly rejected bad schemas)

Total schemas checked: 110
    

Precision vs Recall tradeoff with concrete examples

Precision here means how many schemas accepted as correct truly are correct. High precision avoids using bad schemas that cause errors.

Recall means how many of all correct schemas are accepted. High recall ensures no good schema is missed, allowing full tool use.

Example: If you want to avoid tool failures, prioritize precision to reject bad schemas. But if you want to allow all valid tools, prioritize recall.

What "good" vs "bad" metric values look like for this use case

• Good: Precision > 0.95, Recall > 0.90 -- schemas are mostly correct and few valid ones are missed.
• Bad: Precision < 0.70, Recall < 0.60 -- many bad schemas accepted or many good schemas rejected, causing tool misuse or loss.

Metrics pitfalls

• Accuracy paradox: If most schemas are correct, accuracy can be high even if bad schemas are accepted.
• Data leakage: Using test schemas in training can inflate metrics falsely.
• Overfitting: Schema validation rules too strict may reject valid schemas, lowering recall.

Self-check question

Your schema validation model has 98% accuracy but only 12% recall on correct schemas. Is it good for production? Why or why not?

Answer: No, it is not good. Although accuracy is high, the very low recall means most valid schemas are rejected. This limits tool usability and causes many good tools to be ignored.

Key Result

High precision and recall in schema validation ensure tools are used correctly without missing valid schemas.

Practice

1. What is the main purpose of defining a tool schema in agentic AI?

easy

A. To describe what the tool does and what inputs it needs

B. To write the tool's source code

C. To train the AI model

D. To store user data securely

Solution

1. Step 1: Understand the role of tool schemas

   Tool schemas explain what a tool does and what inputs it requires for proper use.

2. Step 2: Differentiate from other tasks

   Writing code, training models, or storing data are separate tasks not covered by schemas.

3. Final Answer:

   To describe what the tool does and what inputs it needs -> Option A

4. Quick Check:

   Tool schema = tool description + inputs [OK]

Hint: Schemas explain tool purpose and inputs clearly [OK]

Common Mistakes:

• Confusing schema with code implementation
• Thinking schema trains the AI
• Assuming schema stores data

2. Which of the following is the correct way to define a tool schema description in JSON format?

easy

A. {"name": search, "desc": "Finds info", inputs: {query: string}}

B. {"tool": search, description: Finds info, inputs: query string}

C. {"name": "search", "description": "Finds information", "inputs": {"query": "string"}}

D. {name: "search", description: "Finds information", inputs: {query: string}}

Solution

1. Step 1: Check JSON syntax correctness

   {"name": "search", "description": "Finds information", "inputs": {"query": "string"}} uses proper JSON with keys and string values quoted correctly.

2. Step 2: Verify key names and structure

   {"name": "search", "description": "Finds information", "inputs": {"query": "string"}} uses "name", "description", and "inputs" keys properly with correct value types.

3. Final Answer:

   {"name": "search", "description": "Finds information", "inputs": {"query": "string"}} -> Option C

4. Quick Check:

   Valid JSON with quoted keys and strings = {"name": "search", "description": "Finds information", "inputs": {"query": "string"}} [OK]

Hint: Look for quoted keys and string values in JSON [OK]

Common Mistakes:

• Missing quotes around keys or strings
• Using unquoted identifiers
• Incorrect key names or structure

3. Given this tool schema snippet:

{"name": "calculator", "description": "Performs math operations", "inputs": {"operation": "string", "x": "number", "y": "number"}}

What would be the expected output if the tool is called with inputs {"operation": "add", "x": 5, "y": 3}?

medium

A. 8

B. "add"

C. 53

D. Error: missing inputs

Solution

1. Step 1: Understand the tool schema inputs

   The tool expects an operation (like add) and two numbers x and y.

2. Step 2: Apply the operation to inputs

   Adding 5 and 3 results in 8 as output.

3. Final Answer:

   8 -> Option A

4. Quick Check:

   5 + 3 = 8 [OK]

Hint: Match inputs to operation and compute result [OK]

Common Mistakes:

• Returning operation string instead of result
• Concatenating numbers as strings
• Assuming missing inputs cause error

4. You have this tool schema:

{"name": "translator", "description": "Translates text", "inputs": {"text": "string", "target_lang": "string"}}

Which error is present in this schema?

medium

A. Description is too long

B. Name key is missing

C. Input types should be numbers

D. Missing closing brace for inputs object

Solution

1. Step 1: Check JSON syntax carefully

   The inputs object is opened but not closed with a } before the schema ends.

2. Step 2: Verify other keys

   Name key exists, description length is fine, input types as strings are correct.

3. Final Answer:

   Missing closing brace for inputs object -> Option D

4. Quick Check:

   Unclosed braces cause JSON errors [OK]

Hint: Count opening and closing braces carefully [OK]

Common Mistakes:

• Ignoring missing braces
• Thinking input types must be numbers
• Assuming description length matters

5. You want to define a tool schema for a weather forecast tool that takes a city name and date, and returns temperature and conditions. Which schema correctly describes this tool?

hard

A. {"name": "weather", "description": "Forecast tool", "inputs": {"city": "string"}, "outputs": {"temp": "string", "cond": "string"}}

B. {"name": "weather", "description": "Provides weather forecast", "inputs": {"city": "string", "date": "string"}, "outputs": {"temperature": "number", "conditions": "string"}}

C. {"name": "weather", "description": "Gives weather", "inputs": {"location": "string", "day": "date"}}

D. {"name": "weather", "inputs": {"city": "string", "date": "string"}}

Solution

1. Step 1: Check completeness of schema

   {"name": "weather", "description": "Provides weather forecast", "inputs": {"city": "string", "date": "string"}, "outputs": {"temperature": "number", "conditions": "string"}} includes name, description, inputs (city and date), and outputs (temperature and conditions) as required.

2. Step 2: Verify input and output types

   Inputs are strings (city, date), outputs are number and string, matching expected data.

3. Final Answer:

   {"name": "weather", "description": "Provides weather forecast", "inputs": {"city": "string", "date": "string"}, "outputs": {"temperature": "number", "conditions": "string"}} -> Option B

4. Quick Check:

   Complete schema with inputs and outputs = {"name": "weather", "description": "Provides weather forecast", "inputs": {"city": "string", "date": "string"}, "outputs": {"temperature": "number", "conditions": "string"}} [OK]

Hint: Include inputs and outputs with correct types [OK]

Common Mistakes:

• Omitting outputs section
• Using wrong input/output keys or types
• Leaving out description