Defining tool schemas and descriptions in Agentic AI - Model Metrics & Evaluation

Metrics & Evaluation - Defining tool schemas and descriptions

Which metric matters for Defining tool schemas and descriptions and WHY

When defining tool schemas and descriptions, the key metric is accuracy of the schema in representing the tool's capabilities and constraints. This ensures the tool behaves as expected when used by AI agents. Accuracy here means the schema correctly captures all required inputs, outputs, and usage rules without errors.

Why accuracy? Because an incorrect schema can cause the AI to misuse the tool, leading to wrong results or failures. Clear, precise descriptions help the AI understand how to call the tool properly.

Confusion matrix or equivalent visualization

Schema Validation Confusion Matrix (Example):

               | Correct Schema | Incorrect Schema
---------------|----------------|-----------------
Predicted Good |       90       |        5        
Predicted Bad  |       3        |       12        

- True Positives (TP): 90 (correctly identified good schemas)
- False Positives (FP): 5 (incorrectly accepted bad schemas)
- False Negatives (FN): 3 (missed good schemas)
- True Negatives (TN): 12 (correctly rejected bad schemas)

Total schemas checked: 110
    

Precision vs Recall tradeoff with concrete examples

Precision here means how many schemas accepted as correct truly are correct. High precision avoids using bad schemas that cause errors.

Recall means how many of all correct schemas are accepted. High recall ensures no good schema is missed, allowing full tool use.

Example: If you want to avoid tool failures, prioritize precision to reject bad schemas. But if you want to allow all valid tools, prioritize recall.

What "good" vs "bad" metric values look like for this use case

• Good: Precision > 0.95, Recall > 0.90 -- schemas are mostly correct and few valid ones are missed.
• Bad: Precision < 0.70, Recall < 0.60 -- many bad schemas accepted or many good schemas rejected, causing tool misuse or loss.

Metrics pitfalls

• Accuracy paradox: If most schemas are correct, accuracy can be high even if bad schemas are accepted.
• Data leakage: Using test schemas in training can inflate metrics falsely.
• Overfitting: Schema validation rules too strict may reject valid schemas, lowering recall.

Self-check question

Your schema validation model has 98% accuracy but only 12% recall on correct schemas. Is it good for production? Why or why not?

Answer: No, it is not good. Although accuracy is high, the very low recall means most valid schemas are rejected. This limits tool usability and causes many good tools to be ignored.

Key Result

High precision and recall in schema validation ensure tools are used correctly without missing valid schemas.