Data extraction from text in Prompt Engineering / GenAI - Model Metrics & Evaluation

Metrics & Evaluation - Data extraction from text

Which metric matters for Data extraction from text and WHY

For data extraction from text, the key metrics are Precision and Recall. Precision tells us how many extracted pieces of information are actually correct. Recall tells us how many of the correct pieces of information were found by the model. We want both to be high because extracting wrong data (low precision) or missing important data (low recall) both cause problems.

Confusion Matrix Example

      | Extracted Correct | Extracted Incorrect |
      |-------------------|--------------------|
      | True Positives (TP) = 80 | False Positives (FP) = 20 |
      | False Negatives (FN) = 15 | True Negatives (TN) = N/A |

      Total samples = TP + FP + FN = 115 (TN not used here)

      Precision = TP / (TP + FP) = 80 / (80 + 20) = 0.80
      Recall = TP / (TP + FN) = 80 / (80 + 15) = 0.8421
    

Precision vs Recall Tradeoff with Examples

If the model extracts too many pieces of data, it may include wrong ones, lowering precision. For example, extracting phone numbers from text but including random numbers that are not phones.

If the model extracts too few pieces, it may miss important data, lowering recall. For example, missing some email addresses in a document.

In some cases, high precision is more important (e.g., legal documents where wrong data is harmful). In others, high recall is key (e.g., extracting all mentions of symptoms in medical notes).

Good vs Bad Metric Values for Data Extraction

• Good: Precision and Recall both above 0.85 means most extracted data is correct and most correct data is found.
• Bad: Precision below 0.5 means many wrong extractions. Recall below 0.5 means many missed extractions.
• High precision but very low recall means model is too strict, missing data.
• High recall but very low precision means model extracts too much noise.

Common Pitfalls in Metrics for Data Extraction

• Accuracy paradox: Accuracy can be misleading if most text does not contain extractable data. High accuracy can happen by mostly predicting "no data".
• Data leakage: If test data is too similar to training data, metrics look better than real-world performance.
• Overfitting: Model performs well on training data but poorly on new text, causing misleading high metrics during training.
• Ignoring partial matches: Sometimes extracted data is close but not exact. Metrics should consider partial credit if relevant.

Self Check

Your data extraction model has 98% accuracy but only 12% recall on key information. Is it good for production?

Answer: No. The model misses most of the important data (low recall), even if it rarely extracts wrong data (high accuracy). This means it will fail to find most needed information, so it is not ready for production.

Key Result

Precision and recall are key to measure how well data extraction finds correct information without missing or adding errors.

Practice

1. What is the main goal of data extraction from text in AI?

easy

A. To find and pull out useful information like names and dates from text

B. To translate text from one language to another

C. To generate new text based on a prompt

D. To compress text files to save space

Solution

1. Step 1: Understand the purpose of data extraction

   Data extraction means finding specific useful info inside text, such as names, dates, or places.

2. Step 2: Compare options to the definition

   Only To find and pull out useful information like names and dates from text matches this purpose exactly, while others describe different tasks like translation or compression.

3. Final Answer:

   To find and pull out useful information like names and dates from text -> Option A

4. Quick Check:

   Data extraction = find useful info [OK]

Hint: Look for the option about finding info inside text [OK]

Common Mistakes:

• Confusing extraction with translation
• Thinking extraction means generating new text
• Mixing extraction with file compression

2. Which of the following is the correct way to call a function extract_entities with a text input doc in Python?

easy

A. extract_entities = doc()

B. extract_entities(doc)

C. extract_entities.doc()

D. extract_entities->doc()

Solution

1. Step 1: Recall Python function call syntax

   In Python, to call a function with an argument, use function_name(argument).

2. Step 2: Check each option

   extract_entities(doc) uses correct syntax: extract_entities(doc). Options A, C, and D are invalid Python syntax for calling a function.

3. Final Answer:

   extract_entities(doc) -> Option B

4. Quick Check:

   Function call = function_name(argument) [OK]

Hint: Remember Python calls use parentheses with arguments inside [OK]

Common Mistakes:

• Using dot notation to call a function
• Assigning function call to function name
• Using arrow notation like other languages

3. Given this Python code using a simple extraction model:

text = "Alice met Bob on 2023-04-01 in Paris."
entities = extract_entities(text)
print(entities)

If extract_entities returns a list of tuples with (entity, type), what is the expected output?

medium

A. {'Alice': 'PERSON', 'Bob': 'PERSON', '2023-04-01': 'DATE', 'Paris': 'LOCATION'}

B. ['Alice', 'Bob', '2023-04-01', 'Paris']

C. None

D. [('Alice', 'PERSON'), ('Bob', 'PERSON'), ('2023-04-01', 'DATE'), ('Paris', 'LOCATION')]

Solution

1. Step 1: Understand the function output format

   The function returns a list of tuples, each tuple has (entity, type).

2. Step 2: Match output to expected format

   [('Alice', 'PERSON'), ('Bob', 'PERSON'), ('2023-04-01', 'DATE'), ('Paris', 'LOCATION')] matches a list of tuples with entity and type pairs. ['Alice', 'Bob', '2023-04-01', 'Paris'] is just a list of strings, A is a dictionary, and D is None.

3. Final Answer:

   [('Alice', 'PERSON'), ('Bob', 'PERSON'), ('2023-04-01', 'DATE'), ('Paris', 'LOCATION')] -> Option D

4. Quick Check:

   List of (entity, type) tuples = [('Alice', 'PERSON'), ('Bob', 'PERSON'), ('2023-04-01', 'DATE'), ('Paris', 'LOCATION')] [OK]

Hint: Look for list of tuples format with entity and type [OK]

Common Mistakes:

• Confusing list of strings with list of tuples
• Expecting dictionary instead of list
• Assuming function returns None

4. You have this code snippet:

def extract_entities(text):
    entities = []
    for word in text.split():
        if word.istitle():
            entities.append((word, 'PERSON'))
    return entities

text = "John and Mary went to London."
print(extract_entities(text))

What is the bug in this code for extracting entities?

medium

A. It only detects words starting with uppercase, missing multi-word names

B. It does not split text into words

C. It returns a string instead of a list

D. It crashes because of missing import

Solution

1. Step 1: Analyze the extraction logic

   The code checks if each word starts with uppercase (istitle) and labels it as 'PERSON'.

2. Step 2: Identify limitation

   This misses multi-word names like 'New York' or full names with multiple words. It only detects single capitalized words.

3. Final Answer:

   It only detects words starting with uppercase, missing multi-word names -> Option A

4. Quick Check:

   Single-word detection limitation = It only detects words starting with uppercase, missing multi-word names [OK]

Hint: Check if code handles multi-word names or just single words [OK]

Common Mistakes:

• Thinking split() is missing
• Assuming return type is wrong
• Expecting import needed for this code

5. You want to extract dates and locations from a large text using a pretrained AI model. Which approach best improves accuracy and speed?

hard

A. Use a generic language model without any fine-tuning

B. Manually write rules to find dates and locations using string matching

C. Use a named entity recognition (NER) model fine-tuned on your domain data

D. Extract all capitalized words as locations and all numbers as dates

Solution

1. Step 1: Consider model choice for extraction

   Fine-tuning a NER model on your specific domain helps it learn patterns and improves accuracy.

2. Step 2: Compare other options

   Manual rules are slow and brittle, generic models lack domain knowledge, and simple heuristics miss many cases.

3. Final Answer:

   Use a named entity recognition (NER) model fine-tuned on your domain data -> Option C

4. Quick Check:

   Fine-tuned NER model = best accuracy and speed [OK]

Hint: Fine-tune NER models for best extraction results [OK]

Common Mistakes:

• Relying on manual rules only
• Using generic models without tuning
• Using simple heuristics that miss cases