Sandboxing dangerous operations in Agentic AI - Model Metrics & Evaluation

Metrics & Evaluation - Sandboxing dangerous operations

Which metric matters for sandboxing dangerous operations and WHY

When sandboxing dangerous operations, the key metrics are False Positive Rate and False Negative Rate. This is because sandboxing aims to block harmful actions (like running unsafe code) without stopping safe ones.

A False Positive means safe operations are blocked, causing inconvenience or loss of functionality.

A False Negative means dangerous operations slip through, risking security or damage.

Therefore, metrics like Precision (how many blocked operations are truly dangerous) and Recall (how many dangerous operations are caught) are critical to balance safety and usability.

Confusion matrix for sandboxing dangerous operations

      | Predicted Safe       | Predicted Dangerous  |
      |----------------------|---------------------|
      | True Safe (TN)       | False Positive (FP)  |
      | False Negative (FN)  | True Dangerous (TP)  |

      Total operations = TP + FP + TN + FN

      Example:
      TP = 90 (dangerous correctly blocked)
      FP = 10 (safe wrongly blocked)
      TN = 890 (safe correctly allowed)
      FN = 10 (dangerous wrongly allowed)
    

Precision vs Recall tradeoff with examples

If the sandbox blocks too many operations, it has high precision but low recall. This means it rarely blocks safe actions (few false positives), but misses many dangerous ones (many false negatives). This risks security.

If the sandbox blocks too little, it has high recall but low precision. It catches most dangerous actions but also blocks many safe ones, frustrating users.

Example: A sandbox for running user code in a website should catch all harmful code (high recall) but not block normal code (high precision). Balancing these avoids security risks and user frustration.

What "good" vs "bad" metric values look like for sandboxing

• Good: Precision > 0.9 and Recall > 0.9 means most dangerous operations are blocked and few safe ones are stopped.
• Bad: Precision < 0.5 means many safe operations are blocked, hurting usability.
• Bad: Recall < 0.5 means many dangerous operations are missed, risking security.
• Balanced: F1 score near 1.0 shows good overall performance.

Common pitfalls in sandboxing metrics

• Accuracy paradox: If dangerous operations are rare, high accuracy can be misleading by mostly predicting safe.
• Data leakage: Testing on data similar to training can inflate metrics, hiding real risks.
• Overfitting: Sandbox rules too strict on training data may fail on new dangerous operations.
• Ignoring false negatives: Missing dangerous operations is often more harmful than blocking safe ones.

Self-check question

Your sandbox model has 98% accuracy but only 12% recall on dangerous operations. Is it good for production? Why or why not?

Answer: No, it is not good. The low recall means it misses 88% of dangerous operations, which is a big security risk. High accuracy is misleading here because most operations are safe, so the model mostly predicts safe and appears accurate.

Key Result

Precision and recall are key to balance blocking dangerous operations while allowing safe ones in sandboxing.

Practice

1. What is the main purpose of sandboxing dangerous operations in agentic AI?

easy

A. To run risky code safely without harming the main system

B. To speed up the execution of all code

C. To permanently delete unsafe files automatically

D. To make the code run without any errors

Solution

1. Step 1: Understand sandboxing concept

   Sandboxing creates a safe space to run risky code separately from the main system.

2. Step 2: Identify the main goal

   The goal is to protect the system from crashes or data loss caused by dangerous operations.

3. Final Answer:

   To run risky code safely without harming the main system -> Option A

4. Quick Check:

   Sandboxing = safe risky code execution [OK]

Hint: Sandboxing isolates risky code to protect your system [OK]

Common Mistakes:

• Thinking sandboxing speeds up all code
• Believing sandboxing deletes files automatically
• Assuming sandboxing removes all errors

2. Which of the following is the correct way to start a sandbox environment in Python using the sandbox module?

easy

A. sandbox.init()

B. sandbox.run()

C. sandbox.execute()

D. sandbox.start()

Solution

1. Step 1: Recall sandbox module usage

   The common method to begin a sandbox session is start() in many sandbox libraries.

2. Step 2: Match method names

   Among the options, only sandbox.start() correctly initiates the sandbox environment.

3. Final Answer:

   sandbox.start() -> Option D

4. Quick Check:

   Start sandbox = sandbox.start() [OK]

Hint: Look for 'start' to begin sandbox safely [OK]

Common Mistakes:

• Using run() which may execute outside sandbox
• Confusing execute() with start()
• Using init() which may not start sandbox

3. Consider this Python code snippet using a sandbox to run a risky operation:

import sandbox
sandbox.start()
result = sandbox.run('2 + 2')
sandbox.stop()
print(result)

What will be printed?

medium

A. 4

B. '2 + 2'

C. Error: sandbox.run not defined

D. None

Solution

1. Step 1: Understand sandbox.run behavior

   The sandbox.run method executes the string expression safely inside the sandbox.

2. Step 2: Evaluate the expression '2 + 2'

   Evaluating '2 + 2' returns the integer 4, which is stored in result.

3. Final Answer:

   4 -> Option A

4. Quick Check:

   Sandbox runs code safely, output = 4 [OK]

Hint: Sandbox.run executes string code safely and returns result [OK]

Common Mistakes:

• Thinking sandbox.run returns the string itself
• Assuming sandbox.run is undefined
• Expecting None instead of result

4. You wrote this code to sandbox a dangerous file operation:

import sandbox
sandbox.start()
open('/etc/passwd', 'w').write('hacked')
sandbox.stop()

But the file was overwritten on your system. What is the likely error?

medium

A. The sandbox.stop() was missing

B. Sandbox was not properly isolating file writes

C. The open function is blocked in sandbox

D. The code should use sandbox.write() instead

Solution

1. Step 1: Analyze sandbox isolation failure

   If the file was overwritten, sandbox did not isolate the file write operation properly.

2. Step 2: Check other options

   Stopping sandbox does not affect isolation during execution; open is not necessarily blocked; sandbox.write() is not a standard method.

3. Final Answer:

   Sandbox was not properly isolating file writes -> Option B

4. Quick Check:

   Sandbox isolation failure = file overwritten [OK]

Hint: Check if sandbox truly isolates file operations [OK]

Common Mistakes:

• Assuming stopping sandbox fixes isolation
• Thinking open() is always blocked
• Expecting sandbox.write() method exists

5. You want to safely run user-submitted Python code that may contain dangerous operations like file access or network calls. Which approach best uses sandboxing to protect your system?

hard

A. Run the code on your main system but monitor CPU usage

B. Run the code directly with exec() and catch exceptions

C. Run the code inside a containerized sandbox limiting file and network access

D. Run the code after removing all import statements manually

Solution

1. Step 1: Understand sandboxing for dangerous code

   Containerized sandboxing isolates code with strict limits on file and network access.

2. Step 2: Evaluate other options

   Using exec() directly is unsafe; monitoring CPU does not prevent damage; manual import removal is error-prone and incomplete.

3. Final Answer:

   Run the code inside a containerized sandbox limiting file and network access -> Option C

4. Quick Check:

   Container sandbox = safest isolation [OK]

Hint: Use container sandbox to limit risky operations [OK]

Common Mistakes:

• Trusting exec() without isolation
• Relying on CPU monitoring only
• Trying manual code cleaning for safety