Data extraction from text in Prompt Engineering / GenAI - Model Metrics & Evaluation

Metrics & Evaluation - Data extraction from text

Which metric matters for Data extraction from text and WHY

For data extraction from text, the key metrics are Precision and Recall. Precision tells us how many extracted pieces of information are actually correct. Recall tells us how many of the correct pieces of information were found by the model. We want both to be high because extracting wrong data (low precision) or missing important data (low recall) both cause problems.

Confusion Matrix Example

      | Extracted Correct | Extracted Incorrect |
      |-------------------|--------------------|
      | True Positives (TP) = 80 | False Positives (FP) = 20 |
      | False Negatives (FN) = 15 | True Negatives (TN) = N/A |

      Total samples = TP + FP + FN = 115 (TN not used here)

      Precision = TP / (TP + FP) = 80 / (80 + 20) = 0.80
      Recall = TP / (TP + FN) = 80 / (80 + 15) = 0.8421
    

Precision vs Recall Tradeoff with Examples

If the model extracts too many pieces of data, it may include wrong ones, lowering precision. For example, extracting phone numbers from text but including random numbers that are not phones.

If the model extracts too few pieces, it may miss important data, lowering recall. For example, missing some email addresses in a document.

In some cases, high precision is more important (e.g., legal documents where wrong data is harmful). In others, high recall is key (e.g., extracting all mentions of symptoms in medical notes).

Good vs Bad Metric Values for Data Extraction

• Good: Precision and Recall both above 0.85 means most extracted data is correct and most correct data is found.
• Bad: Precision below 0.5 means many wrong extractions. Recall below 0.5 means many missed extractions.
• High precision but very low recall means model is too strict, missing data.
• High recall but very low precision means model extracts too much noise.

Common Pitfalls in Metrics for Data Extraction

• Accuracy paradox: Accuracy can be misleading if most text does not contain extractable data. High accuracy can happen by mostly predicting "no data".
• Data leakage: If test data is too similar to training data, metrics look better than real-world performance.
• Overfitting: Model performs well on training data but poorly on new text, causing misleading high metrics during training.
• Ignoring partial matches: Sometimes extracted data is close but not exact. Metrics should consider partial credit if relevant.

Self Check

Your data extraction model has 98% accuracy but only 12% recall on key information. Is it good for production?

Answer: No. The model misses most of the important data (low recall), even if it rarely extracts wrong data (high accuracy). This means it will fail to find most needed information, so it is not ready for production.

Key Result

Precision and recall are key to measure how well data extraction finds correct information without missing or adding errors.