Sandboxing dangerous operations in Agentic Ai - Model Metrics & Evaluation

Metrics & Evaluation - Sandboxing dangerous operations

Which metric matters for sandboxing dangerous operations and WHY

When sandboxing dangerous operations, the key metrics are False Positive Rate and False Negative Rate. This is because sandboxing aims to block harmful actions (like running unsafe code) without stopping safe ones.

A False Positive means safe operations are blocked, causing inconvenience or loss of functionality.

A False Negative means dangerous operations slip through, risking security or damage.

Therefore, metrics like Precision (how many blocked operations are truly dangerous) and Recall (how many dangerous operations are caught) are critical to balance safety and usability.

Confusion matrix for sandboxing dangerous operations

      | Predicted Safe       | Predicted Dangerous  |
      |----------------------|---------------------|
      | True Safe (TN)       | False Positive (FP)  |
      | False Negative (FN)  | True Dangerous (TP)  |

      Total operations = TP + FP + TN + FN

      Example:
      TP = 90 (dangerous correctly blocked)
      FP = 10 (safe wrongly blocked)
      TN = 890 (safe correctly allowed)
      FN = 10 (dangerous wrongly allowed)
    

Precision vs Recall tradeoff with examples

If the sandbox blocks too many operations, it has high precision but low recall. This means it rarely blocks safe actions (few false positives), but misses many dangerous ones (many false negatives). This risks security.

If the sandbox blocks too little, it has high recall but low precision. It catches most dangerous actions but also blocks many safe ones, frustrating users.

Example: A sandbox for running user code in a website should catch all harmful code (high recall) but not block normal code (high precision). Balancing these avoids security risks and user frustration.

What "good" vs "bad" metric values look like for sandboxing

• Good: Precision > 0.9 and Recall > 0.9 means most dangerous operations are blocked and few safe ones are stopped.
• Bad: Precision < 0.5 means many safe operations are blocked, hurting usability.
• Bad: Recall < 0.5 means many dangerous operations are missed, risking security.
• Balanced: F1 score near 1.0 shows good overall performance.

Common pitfalls in sandboxing metrics

• Accuracy paradox: If dangerous operations are rare, high accuracy can be misleading by mostly predicting safe.
• Data leakage: Testing on data similar to training can inflate metrics, hiding real risks.
• Overfitting: Sandbox rules too strict on training data may fail on new dangerous operations.
• Ignoring false negatives: Missing dangerous operations is often more harmful than blocking safe ones.

Self-check question

Your sandbox model has 98% accuracy but only 12% recall on dangerous operations. Is it good for production? Why or why not?

Answer: No, it is not good. The low recall means it misses 88% of dangerous operations, which is a big security risk. High accuracy is misleading here because most operations are safe, so the model mostly predicts safe and appears accurate.

Key Result

Precision and recall are key to balance blocking dangerous operations while allowing safe ones in sandboxing.