Output filtering and safety checks in Agentic Ai - Model Metrics & Evaluation

Metrics & Evaluation - Output filtering and safety checks

Which metric matters for output filtering and safety checks and WHY

For output filtering and safety checks, the key metrics are False Positive Rate and False Negative Rate. False positives mean safe content is wrongly blocked, which hurts user experience. False negatives mean unsafe content is missed, which can cause harm. Balancing these is critical to keep outputs safe without blocking too much useful content.

Confusion matrix for output filtering

          | Predicted Safe | Predicted Unsafe
    ------|----------------|-----------------
    Actual Safe   |      TN=85     |      FP=15
    Actual Unsafe |      FN=10     |      TP=90
    

Here, TP means unsafe content correctly blocked, FP means safe content wrongly blocked, TN means safe content correctly allowed, and FN means unsafe content missed.

Precision vs Recall tradeoff with examples

Precision measures how many blocked outputs are truly unsafe. High precision means few safe outputs are blocked (low false positives).

Recall measures how many unsafe outputs are caught. High recall means few unsafe outputs slip through (low false negatives).

Example: If you block too much (high recall), users get annoyed by safe content blocked (low precision). If you block too little (high precision), unsafe content may appear (low recall). Finding the right balance depends on the use case.

What good vs bad metric values look like

Good: Precision and recall both above 90%, meaning most unsafe content is blocked and most safe content is allowed.

Bad: Precision below 70% means many safe outputs blocked, hurting user trust. Recall below 50% means many unsafe outputs missed, risking harm.

Common pitfalls in output filtering metrics

• Accuracy paradox: If unsafe content is rare, a model blocking nothing can have high accuracy but be useless.
• Data leakage: If test data leaks into training, metrics look better than real performance.
• Overfitting: Model blocks training unsafe content well but fails on new unsafe content.

Self-check question

Your output filter has 98% accuracy but only 12% recall on unsafe content. Is it good for production? Why or why not?

Answer: No, it is not good. The low recall means it misses 88% of unsafe content, which can cause harm. High accuracy is misleading because most content is safe, so blocking nothing looks accurate but unsafe outputs slip through.

Key Result

Balancing false positives and false negatives is key to effective output filtering and safety checks.