PII detection and redaction in Prompt Engineering / GenAI - Model Metrics & Evaluation

Metrics & Evaluation - PII detection and redaction

Which metric matters for PII detection and redaction and WHY

For PII detection, Recall is very important because we want to find as many personal details as possible to protect privacy. Missing a PII means sensitive data leaks. Precision also matters because marking too many words as PII causes unnecessary redaction, making text hard to read. So, we balance both using the F1 score, which combines precision and recall into one number.

Confusion matrix for PII detection

      | Predicted PII | Predicted Non-PII |
      |---------------|-------------------|
      | True Positive (TP)  | False Positive (FP) |
      | False Negative (FN) | True Negative (TN)  |

      Example:
      TP = 80 (correctly found PII)
      FP = 10 (wrongly marked non-PII as PII)
      FN = 20 (missed PII)
      TN = 890 (correctly ignored non-PII)

      Total samples = 80 + 10 + 20 + 890 = 1000
    

From this, we calculate:

• Precision = 80 / (80 + 10) = 0.89
• Recall = 80 / (80 + 20) = 0.80
• F1 score = 2 * (0.89 * 0.80) / (0.89 + 0.80) ≈ 0.84

Precision vs Recall tradeoff with examples

If we focus too much on precision, we only mark PII when very sure. This means fewer false alarms but we might miss some PII (low recall). For example, a system that only redacts very obvious phone numbers but misses nicknames or emails.

If we focus too much on recall, we catch almost all PII but also mark many normal words as PII (low precision). This makes the text hard to read because too many words are redacted.

Good PII detection balances both. For example, a system that finds 90% of PII (high recall) and keeps false alarms below 10% (high precision).

What good vs bad metric values look like for PII detection

• Good: Precision ≥ 0.85, Recall ≥ 0.85, F1 ≥ 0.85. This means most PII is found and few false redactions.
• Bad: Precision < 0.5 or Recall < 0.5. This means many false alarms or many missed PII, both harmful.
• Accuracy is less useful here because most text is non-PII, so a model that marks nothing can have high accuracy but is useless.

Common pitfalls in PII detection metrics

• Accuracy paradox: Since most text is non-PII, a model that never detects PII can have high accuracy but zero recall.
• Data leakage: If test data contains PII seen during training, metrics look better but model fails on new data.
• Overfitting: Model memorizes specific PII patterns but misses new types, causing low recall in real use.
• Ignoring context: Some words are PII only in certain contexts; metrics must consider this to avoid false positives.

Self-check question

Your PII detection model has 98% accuracy but only 12% recall on PII. Is it good for production? Why or why not?

Answer: No, it is not good. The high accuracy is misleading because most text is non-PII. The very low recall means it misses 88% of PII, risking privacy leaks. For PII detection, recall must be high to protect sensitive data.

Key Result

High recall is critical to catch most PII, balanced with precision to avoid excessive false redactions.