Prompt injection attacks in Prompt Engineering / GenAI - Model Metrics & Evaluation

Metrics & Evaluation - Prompt injection attacks

Which metric matters for prompt injection attacks and WHY

For prompt injection attacks, the key metric is attack success rate. This measures how often an attacker can trick the AI into following harmful or unintended instructions. A low attack success rate means the AI resists manipulation well. We also look at false positive rate to ensure the AI does not wrongly block safe prompts. Balancing these helps keep the AI both safe and useful.

Confusion matrix for prompt injection detection

          | Predicted Safe | Predicted Attack
    ------|----------------|-----------------
    Safe  |      TN=850    |      FP=50      
    Attack|      FN=30     |      TP=70      

    Total samples = 1000

    Precision = TP / (TP + FP) = 70 / (70 + 50) = 0.58
    Recall = TP / (TP + FN) = 70 / (70 + 30) = 0.70
    

This shows the model catches 70% of attacks (recall) but sometimes flags safe prompts wrongly (false positives).

Precision vs Recall tradeoff with examples

In prompt injection detection:

• High precision means when the AI says a prompt is an attack, it usually is. This avoids blocking good users unfairly.
• High recall means the AI catches most attacks, reducing risk of harmful outputs.

Example: If you want to keep users happy, prioritize precision to avoid false alarms. If safety is critical, prioritize recall to catch more attacks, even if some safe prompts get blocked.

What good vs bad metric values look like

Good values:

• Attack success rate below 5% (low chance attacker tricks AI)
• Precision above 80% (few false alarms)
• Recall above 75% (most attacks caught)

Bad values:

• Attack success rate above 30% (many attacks succeed)
• Precision below 50% (many safe prompts blocked)
• Recall below 40% (most attacks missed)

Common pitfalls in metrics for prompt injection attacks

• Ignoring context: Metrics may look good on test data but fail on new attack types.
• Data leakage: If attack examples leak into training, metrics overestimate real safety.
• Overfitting: Model may memorize known attacks but miss new ones, inflating recall.
• Accuracy paradox: High overall accuracy can hide poor attack detection if attacks are rare.

Self-check question

Your prompt injection detection model has 98% accuracy but only 12% recall on attacks. Is it good for production? Why or why not?

Answer: No, it is not good. The model misses 88% of attacks (low recall), so many harmful prompts get through. High accuracy is misleading because attacks are rare, so the model mostly predicts safe prompts correctly but fails at catching attacks.

Key Result

Attack success rate and recall are key to measure how well prompt injection attacks are detected and blocked.