Information extraction patterns in NLP - Model Metrics & Evaluation

Metrics & Evaluation - Information extraction patterns

Which metric matters for Information Extraction Patterns and WHY

Information extraction (IE) aims to find specific pieces of information from text, like names or dates. The key metrics are Precision and Recall. Precision tells us how many extracted items are actually correct. Recall tells us how many of the total correct items we found. We want both high, but sometimes one matters more depending on the task.

Confusion Matrix for Information Extraction

      | Predicted Yes | Predicted No  |
      |---------------|--------------|
      | True Positive | False Negative|
      | False Positive| True Negative |

    TP = Correctly extracted info
    FP = Extracted info that is wrong
    FN = Missed info that should be extracted
    TN = Correctly ignored non-info
    

Precision vs Recall Tradeoff with Examples

If you want to avoid wrong info in your output, focus on high precision. For example, a legal document extractor must not add false facts.

If you want to find all possible info, even if some are wrong, focus on high recall. For example, a news aggregator wants to catch all names mentioned, even if some are mistakes.

Balancing both is key. The F1 score helps measure this balance.

Good vs Bad Metric Values for IE Patterns

Good: Precision and Recall above 0.8 means most extracted info is correct and most info is found.

Bad: Precision below 0.5 means many wrong extractions. Recall below 0.5 means many missed extractions.

Example: Precision=0.9, Recall=0.85 is good. Precision=0.4, Recall=0.3 is bad.

Common Pitfalls in IE Metrics

• Accuracy paradox: High accuracy can be misleading if most text has no info to extract.
• Data leakage: Testing on data too similar to training inflates metrics.
• Overfitting: Model extracts perfectly on training but fails on new text.
• Ignoring class imbalance: Info to extract is rare, so metrics must consider this.

Self Check

Your IE model has 98% accuracy but only 12% recall on extracting names. Is it good?

Answer: No. The model misses most names (low recall), so it is not useful despite high accuracy. It finds very few correct names.

Key Result

Precision and Recall are key metrics for information extraction patterns, measuring correctness and completeness of extracted data.

Practice

1. What is the main purpose of information extraction patterns in NLP?

easy

A. To automatically find specific facts like names or dates in text

B. To translate text from one language to another

C. To generate new sentences from given words

D. To summarize long documents into short paragraphs

Solution

1. Step 1: Understand the role of information extraction patterns

   These patterns are designed to locate specific pieces of information such as names, dates, or places within text automatically.

2. Step 2: Compare with other NLP tasks

   Translation, generation, and summarization are different NLP tasks and do not focus on extracting facts.

3. Final Answer:

   To automatically find specific facts like names or dates in text -> Option A

4. Quick Check:

   Information extraction = find facts [OK]

Hint: Patterns extract facts, not translate or summarize [OK]

Common Mistakes:

• Confusing extraction with translation
• Thinking patterns generate new text
• Mixing extraction with summarization

2. Which of the following is a correct example of a simple pattern to extract dates in text?

easy

A. \b[A-Z]{2,}\b (matches uppercase words)

B. \b\d{4}-\d{2}-\d{2}\b (matches YYYY-MM-DD format)

C. \w+@\w+\.com (matches email addresses)

D. \d+\s+\w+ (matches any number followed by a word)

Solution

1. Step 1: Identify the pattern for dates

   The pattern \b\d{4}-\d{2}-\d{2}\b matches a 4-digit year, 2-digit month, and 2-digit day separated by dashes, which is a common date format.

2. Step 2: Check other options

   \d+\s+\w+ (matches any number followed by a word) matches number + word but is too general; C matches emails; A matches uppercase words, not dates.

3. Final Answer:

   \b\d{4}-\d{2}-\d{2}\b (matches YYYY-MM-DD format) -> Option B

4. Quick Check:

   Date pattern = \b\d{4}-\d{2}-\d{2}\b (matches YYYY-MM-DD format) [OK]

Hint: Look for year-month-day format in regex [OK]

Common Mistakes:

• Choosing patterns that match emails or words instead of dates
• Ignoring word boundaries \b in regex
• Confusing number patterns with date formats

3. Given this pattern to extract person names: \b(Mr|Ms|Dr)\.\s+[A-Z][a-z]+\b, what will be the output when applied to the text: "Dr. Smith and Mr. Johnson went to the park."?

medium

A. ["Dr", "Mr"]

B. ["Smith", "Johnson"]

C. ["Dr. Smith", "Mr. Johnson"]

D. [] (empty list)

Solution

1. Step 1: Understand the regex pattern

   The pattern matches titles (Mr, Ms, Dr) followed by a dot, a space, and a capitalized last name.

2. Step 2: Apply pattern to the text

   In the text, "Dr. Smith" and "Mr. Johnson" both match the pattern exactly.

3. Final Answer:

   ["Dr. Smith", "Mr. Johnson"] -> Option C

4. Quick Check:

   Pattern matches title + name = ["Dr. Smith", "Mr. Johnson"] [OK]

Hint: Match title + dot + space + capitalized name [OK]

Common Mistakes:

• Extracting only last names without titles
• Extracting only titles without names
• Getting empty results due to pattern mismatch

4. Identify the error in this pattern meant to extract email addresses: \b[\w.-]+@[\w.-]+\b

medium

A. It misses the domain extension like .com or .org

B. It uses incorrect character classes for emails

C. It does not match the '@' symbol

D. It matches only uppercase letters

Solution

1. Step 1: Analyze the pattern components

   The pattern matches word characters, dots, or dashes before and after '@', but stops at word boundary without requiring domain extensions like '.com'.

2. Step 2: Identify missing part

   Valid emails usually end with a domain extension (e.g., '.com'), which this pattern does not enforce, so it may match incomplete emails.

3. Final Answer:

   It misses the domain extension like .com or .org -> Option A

4. Quick Check:

   Email pattern missing domain extension = It misses the domain extension like .com or .org [OK]

Hint: Check if pattern includes domain extensions like .com [OK]

Common Mistakes:

• Assuming '@' is not matched
• Thinking character classes are wrong
• Ignoring domain extension importance

5. You want to extract locations from text using patterns that match city names followed by state abbreviations, like "Austin TX" or "Denver CO". Which pattern best fits this task?

hard

A. \b\w+@\w+\.com\b (email addresses)

B. \b\d{5}\b (five digit numbers)

C. \b[A-Z]{2,}\b (two or more uppercase letters only)

D. \b[A-Z][a-z]+\s+[A-Z]{2}\b (capitalized city name + space + two uppercase letters)

Solution

1. Step 1: Understand the location format

   Locations are city names starting with a capital letter followed by a two-letter uppercase state abbreviation.

2. Step 2: Match pattern to format

   Pattern \b[A-Z][a-z]+\s+[A-Z]{2}\b matches a capitalized word, a space, then exactly two uppercase letters, fitting the example.

3. Final Answer:

   \b[A-Z][a-z]+\s+[A-Z]{2}\b (capitalized city name + space + two uppercase letters) -> Option D

4. Quick Check:

   City + state abbreviation pattern = \b[A-Z][a-z]+\s+[A-Z]{2}\b (capitalized city name + space + two uppercase letters) [OK]

Hint: City capitalized + space + 2 uppercase letters [OK]

Common Mistakes:

• Choosing patterns for zip codes or emails
• Matching only uppercase words without city name
• Ignoring space between city and state