Building custom tools in Agentic AI - Model Metrics & Evaluation

Metrics & Evaluation - Building custom tools

Which metric matters for Building custom tools and WHY

When building custom AI tools, the key metric depends on the tool's goal. For example, if the tool classifies text, accuracy shows overall correctness. But if the tool detects rare events, recall is vital to catch as many true cases as possible. If the tool must avoid false alarms, precision matters more. Choosing the right metric helps you know if your tool works well for its purpose.

Confusion matrix example

      | Predicted Positive | Predicted Negative |
      |--------------------|--------------------|
      | True Positive (TP): 40 | False Negative (FN): 10 |
      | False Positive (FP): 5 | True Negative (TN): 45 |

      Total samples = 40 + 10 + 5 + 45 = 100

      Precision = TP / (TP + FP) = 40 / (40 + 5) = 0.89
      Recall = TP / (TP + FN) = 40 / (40 + 10) = 0.80
      Accuracy = (TP + TN) / Total = (40 + 45) / 100 = 0.85
      F1 Score = 2 * (Precision * Recall) / (Precision + Recall) = 2 * (0.89 * 0.80) / (0.89 + 0.80) ≈ 0.84
    

Precision vs Recall tradeoff with examples

Imagine building a custom tool to detect spam emails:

• High Precision: The tool marks emails as spam only when very sure. Few good emails get wrongly marked (few false alarms). But it might miss some spam (lower recall).
• High Recall: The tool catches almost all spam emails. But it might mark some good emails as spam (more false alarms).

Choosing between precision and recall depends on what is worse: missing spam or wrongly blocking good emails.

What "good" vs "bad" metric values look like for Building custom tools

Good metrics:

• Accuracy above 85% for balanced tasks
• Precision and recall both above 80% for critical detection tools
• F1 score close to 1 means balanced and strong performance

Bad metrics:

• Accuracy near 50% on balanced data means guessing
• Precision very low (e.g., 30%) means many false alarms
• Recall very low (e.g., 20%) means many misses
• Big gap between precision and recall shows imbalance

Common pitfalls when evaluating custom tools

• Accuracy paradox: High accuracy can be misleading if data is imbalanced (e.g., 95% accuracy but only detecting the majority class).
• Data leakage: Using future or test data during training inflates metrics falsely.
• Overfitting: Very high training accuracy but low test accuracy means the tool learned noise, not real patterns.
• Ignoring metric tradeoffs: Focusing only on accuracy without considering precision or recall can hide problems.

Self-check question

Your custom tool has 98% accuracy but only 12% recall on detecting fraud cases. Is it good for production? Why or why not?

Answer: No, it is not good. The tool misses 88% of fraud cases (low recall), which is dangerous. High accuracy likely comes from many non-fraud cases being correctly identified, but the tool fails at its main goal: catching fraud.

Key Result

Choosing the right metric like precision or recall is key to knowing if your custom AI tool works well for its specific task.