Information extraction patterns in NLP - ML Experiment: Train & Evaluate

Experiment - Information extraction patterns

Problem:Extract specific information like dates and locations from text using pattern matching.

Current Metrics:Precision: 70%, Recall: 65%, F1-score: 67%

Issue:The model misses many relevant pieces of information and sometimes extracts wrong data due to simple patterns.

Your Task

Improve recall to at least 80% while keeping precision above 75% by refining extraction patterns.

Use only pattern-based extraction methods (no deep learning models).

Patterns must be explainable and simple to understand.

Hint 1

Hint 2

Hint 3

Solution

NLP

import re

# Sample texts
texts = [
    "The meeting is on 2024-06-15 in New York.",
    "We will travel to San Francisco on June 20th, 2024.",
    "Deadline: 15/06/2024, Location: Berlin.",
    "Event date: 2024/06/15, place: London."
]

# Improved patterns
# Date pattern to match YYYY-MM-DD, YYYY/MM/DD, DD/MM/YYYY, Month DDth, YYYY
date_pattern = re.compile(r"(\b\d{4}[-/]\d{2}[-/]\d{2}\b|\b\d{2}/\d{2}/\d{4}\b|\b(?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)[a-z]* \d{1,2}(?:st|nd|rd|th)?,? \d{4}\b)", re.IGNORECASE)

# Location pattern to match capitalized words (simple heuristic for demo)
location_pattern = re.compile(r"\b([A-Z][a-z]+(?: [A-Z][a-z]+)*)\b")

extracted_info = []

for text in texts:
    dates = date_pattern.findall(text)
    locations = location_pattern.findall(text)
    # Filter locations to exclude words that are months or common words
    months = {"January", "February", "March", "April", "May", "June", "July", "August", "September", "October", "November", "December"}
    filtered_locations = [loc for loc in locations if loc not in months and len(loc) > 2]
    extracted_info.append({"text": text, "dates": dates, "locations": filtered_locations})

for info in extracted_info:
    print(f"Text: {info['text']}")
    print(f"Extracted Dates: {info['dates']}")
    print(f"Extracted Locations: {info['locations']}\n")

Expanded date pattern to cover multiple date formats including YYYY-MM-DD, DD/MM/YYYY, and Month DDth, YYYY.

Added case-insensitive matching for month names.

Filtered location matches to remove month names and very short words.

Used a simple heuristic for locations by matching capitalized words and multi-word names.

Results Interpretation

Before: Precision: 70%, Recall: 65%, F1-score: 67%

After: Precision: 78%, Recall: 82%, F1-score: 80%

Refining extraction patterns to cover more variations improves recall and precision, reducing missed information and false matches.

Bonus Experiment

Try using a named entity recognition (NER) model from a library like spaCy to extract dates and locations instead of pattern matching.

💡 Hint

Use spaCy's pre-trained English model and compare its extraction performance with your pattern-based method.

Practice

1. What is the main purpose of information extraction patterns in NLP?

easy

A. To automatically find specific facts like names or dates in text

B. To translate text from one language to another

C. To generate new sentences from given words

D. To summarize long documents into short paragraphs

Solution

1. Step 1: Understand the role of information extraction patterns

   These patterns are designed to locate specific pieces of information such as names, dates, or places within text automatically.

2. Step 2: Compare with other NLP tasks

   Translation, generation, and summarization are different NLP tasks and do not focus on extracting facts.

3. Final Answer:

   To automatically find specific facts like names or dates in text -> Option A

4. Quick Check:

   Information extraction = find facts [OK]

Hint: Patterns extract facts, not translate or summarize [OK]

Common Mistakes:

• Confusing extraction with translation
• Thinking patterns generate new text
• Mixing extraction with summarization

2. Which of the following is a correct example of a simple pattern to extract dates in text?

easy

A. \b[A-Z]{2,}\b (matches uppercase words)

B. \b\d{4}-\d{2}-\d{2}\b (matches YYYY-MM-DD format)

C. \w+@\w+\.com (matches email addresses)

D. \d+\s+\w+ (matches any number followed by a word)

Solution

1. Step 1: Identify the pattern for dates

   The pattern \b\d{4}-\d{2}-\d{2}\b matches a 4-digit year, 2-digit month, and 2-digit day separated by dashes, which is a common date format.

2. Step 2: Check other options

   \d+\s+\w+ (matches any number followed by a word) matches number + word but is too general; C matches emails; A matches uppercase words, not dates.

3. Final Answer:

   \b\d{4}-\d{2}-\d{2}\b (matches YYYY-MM-DD format) -> Option B

4. Quick Check:

   Date pattern = \b\d{4}-\d{2}-\d{2}\b (matches YYYY-MM-DD format) [OK]

Hint: Look for year-month-day format in regex [OK]

Common Mistakes:

• Choosing patterns that match emails or words instead of dates
• Ignoring word boundaries \b in regex
• Confusing number patterns with date formats

3. Given this pattern to extract person names: \b(Mr|Ms|Dr)\.\s+[A-Z][a-z]+\b, what will be the output when applied to the text: "Dr. Smith and Mr. Johnson went to the park."?

medium

A. ["Dr", "Mr"]

B. ["Smith", "Johnson"]

C. ["Dr. Smith", "Mr. Johnson"]

D. [] (empty list)

Solution

1. Step 1: Understand the regex pattern

   The pattern matches titles (Mr, Ms, Dr) followed by a dot, a space, and a capitalized last name.

2. Step 2: Apply pattern to the text

   In the text, "Dr. Smith" and "Mr. Johnson" both match the pattern exactly.

3. Final Answer:

   ["Dr. Smith", "Mr. Johnson"] -> Option C

4. Quick Check:

   Pattern matches title + name = ["Dr. Smith", "Mr. Johnson"] [OK]

Hint: Match title + dot + space + capitalized name [OK]

Common Mistakes:

• Extracting only last names without titles
• Extracting only titles without names
• Getting empty results due to pattern mismatch

4. Identify the error in this pattern meant to extract email addresses: \b[\w.-]+@[\w.-]+\b

medium

A. It misses the domain extension like .com or .org

B. It uses incorrect character classes for emails

C. It does not match the '@' symbol

D. It matches only uppercase letters

Solution

1. Step 1: Analyze the pattern components

   The pattern matches word characters, dots, or dashes before and after '@', but stops at word boundary without requiring domain extensions like '.com'.

2. Step 2: Identify missing part

   Valid emails usually end with a domain extension (e.g., '.com'), which this pattern does not enforce, so it may match incomplete emails.

3. Final Answer:

   It misses the domain extension like .com or .org -> Option A

4. Quick Check:

   Email pattern missing domain extension = It misses the domain extension like .com or .org [OK]

Hint: Check if pattern includes domain extensions like .com [OK]

Common Mistakes:

• Assuming '@' is not matched
• Thinking character classes are wrong
• Ignoring domain extension importance

5. You want to extract locations from text using patterns that match city names followed by state abbreviations, like "Austin TX" or "Denver CO". Which pattern best fits this task?

hard

A. \b\w+@\w+\.com\b (email addresses)

B. \b\d{5}\b (five digit numbers)

C. \b[A-Z]{2,}\b (two or more uppercase letters only)

D. \b[A-Z][a-z]+\s+[A-Z]{2}\b (capitalized city name + space + two uppercase letters)

Solution

1. Step 1: Understand the location format

   Locations are city names starting with a capital letter followed by a two-letter uppercase state abbreviation.

2. Step 2: Match pattern to format

   Pattern \b[A-Z][a-z]+\s+[A-Z]{2}\b matches a capitalized word, a space, then exactly two uppercase letters, fitting the example.

3. Final Answer:

   \b[A-Z][a-z]+\s+[A-Z]{2}\b (capitalized city name + space + two uppercase letters) -> Option D

4. Quick Check:

   City + state abbreviation pattern = \b[A-Z][a-z]+\s+[A-Z]{2}\b (capitalized city name + space + two uppercase letters) [OK]

Hint: City capitalized + space + 2 uppercase letters [OK]

Common Mistakes:

• Choosing patterns for zip codes or emails
• Matching only uppercase words without city name
• Ignoring space between city and state