Why NER extracts structured information in NLP - Why Metrics Matter

Metrics & Evaluation - Why NER extracts structured information

Which metric matters for this concept and WHY

For Named Entity Recognition (NER), the key metrics are Precision, Recall, and F1-score. These metrics tell us how well the model finds and labels entities correctly.

Precision shows how many of the entities the model found are actually correct. This matters because we want to avoid wrong information.

Recall shows how many of the real entities the model managed to find. This matters because missing important information can reduce usefulness.

F1-score balances precision and recall, giving a single number to understand overall quality.

Confusion matrix or equivalent visualization (ASCII)

    Entity Prediction Confusion Matrix:

                 Predicted Entity   Predicted Non-Entity
    Actual Entity        TP                 FN
    Actual Non-Entity    FP                 TN

    Where:
    TP = Correctly found entities
    FP = Wrongly labeled entities
    FN = Missed entities
    TN = Correctly ignored non-entities
    

Precision vs Recall tradeoff with concrete examples

If the NER model has high precision but low recall, it means it finds few entities but those are mostly correct. This is good if you want very reliable info but can miss many entities.

If the model has high recall but low precision, it finds most entities but also many wrong ones. This is good if missing entities is bad, but you must handle extra noise.

Example: In medical records, high recall is important to catch all diseases mentioned, even if some mistakes happen. In legal documents, high precision is important to avoid wrong facts.

What "good" vs "bad" metric values look like for this use case

Good NER model: Precision and recall both above 85%, F1-score above 85%. This means it finds most entities correctly and misses few.

Bad NER model: Precision or recall below 50%. This means many wrong entities or many missed entities, making the output unreliable.

Metrics pitfalls

• Ignoring entity boundaries: Partial matches can inflate scores if not measured carefully.
• Data leakage: If test data is too similar to training, metrics look better than real use.
• Imbalanced entity types: Some entities appear more often, skewing overall metrics.
• Overfitting: Very high training scores but low test scores mean the model memorizes instead of generalizing.

Self-check question

Your NER model has 98% accuracy but only 12% recall on person names. Is it good for production? Why not?

Answer: No, it is not good. Accuracy can be misleading because most words are not entities. The very low recall means the model misses almost all person names, which is critical information. So, despite high accuracy, the model fails to extract important structured information.

Key Result

Precision, recall, and F1-score are key to measure how well NER extracts correct and complete structured information.

Practice

1. Why does Named Entity Recognition (NER) extract structured information from text?

easy

A. To translate text into different languages

B. To remove all punctuation from the text

C. To generate random sentences from input text

D. To turn messy text into organized data that machines can understand

Solution

1. Step 1: Understand the purpose of NER

   NER identifies names like people, places, and dates in text.

2. Step 2: Connect NER output to structured data

   By labeling these names, NER turns unorganized text into clear, usable information.

3. Final Answer:

   To turn messy text into organized data that machines can understand -> Option D

4. Quick Check:

   NER = structured data extraction [OK]

Hint: NER organizes text into clear data for machines [OK]

Common Mistakes:

• Thinking NER translates languages
• Believing NER generates new text
• Confusing NER with text cleaning

2. Which of the following is the correct way to describe the output of a NER system?

easy

A. Text with entities labeled as categories like Person or Location

B. A list of sentences without any labels

C. A summary of the input text

D. A translation of the text into code

Solution

1. Step 1: Identify what NER labels

   NER tags parts of text with entity types such as Person, Location, or Organization.

2. Step 2: Match output description

   Output is text with these labels, not just plain sentences or summaries.

3. Final Answer:

   Text with entities labeled as categories like Person or Location -> Option A

4. Quick Check:

   NER output = labeled entities [OK]

Hint: NER output labels entities in text [OK]

Common Mistakes:

• Confusing NER output with summaries
• Thinking NER removes labels
• Assuming NER translates text

3. Given the sentence: "Apple was founded by Steve Jobs in California." What structured information would a NER system most likely extract?

medium

A. {"Apple": "Organization", "Steve Jobs": "Person", "California": "Location"}

B. {"Apple": "Fruit", "Steve Jobs": "Person", "California": "Fruit"}

C. {"Apple": "Person", "Steve Jobs": "Organization", "California": "Location"}

D. {"Apple": "Location", "Steve Jobs": "Location", "California": "Person"}

Solution

1. Step 1: Identify entities in the sentence

   "Apple" is a company (Organization), "Steve Jobs" is a person, and "California" is a place (Location).

2. Step 2: Match entities to correct categories

   Assign correct labels: Apple - Organization, Steve Jobs - Person, California - Location.

3. Final Answer:

   {"Apple": "Organization", "Steve Jobs": "Person", "California": "Location"} -> Option A

4. Quick Check:

   Entities labeled correctly = {"Apple": "Organization", "Steve Jobs": "Person", "California": "Location"} [OK]

Hint: Match names to real-world categories [OK]

Common Mistakes:

• Labeling Apple as a fruit instead of organization
• Swapping person and organization labels
• Mislabeling locations as persons

4. A NER system outputs: {"Paris": "Person", "Eiffel Tower": "Location"}. What is the likely error?

medium

A. NER systems do not label locations

B. The entity "Eiffel Tower" should be labeled as a Person, not a Location

C. The entity "Paris" should be labeled as a Location, not a Person

D. Both entities are correctly labeled

Solution

1. Step 1: Check entity meanings

   "Paris" is a city, so it should be labeled as a Location, not a Person.

2. Step 2: Verify other labels

   "Eiffel Tower" is a landmark, correctly labeled as Location.

3. Final Answer:

   The entity "Paris" should be labeled as a Location, not a Person -> Option C

4. Quick Check:

   Incorrect label for Paris = The entity "Paris" should be labeled as a Location, not a Person [OK]

Hint: Check if entity matches real-world category [OK]

Common Mistakes:

• Accepting wrong labels without question
• Confusing landmarks with people
• Ignoring obvious entity meanings

5. How can NER help improve a chatbot's ability to answer questions about events?

hard

A. By translating user messages into multiple languages automatically

B. By extracting event names, dates, and locations to provide precise answers

C. By generating random responses to confuse users

D. By deleting all user input to reduce processing time

Solution

1. Step 1: Understand chatbot needs

   Chatbots need clear facts like event names, dates, and places to answer well.

2. Step 2: Role of NER in chatbots

   NER extracts these key details from user input, enabling the chatbot to respond accurately.

3. Final Answer:

   By extracting event names, dates, and locations to provide precise answers -> Option B

4. Quick Check:

   NER improves chatbot accuracy = By extracting event names, dates, and locations to provide precise answers [OK]

Hint: NER finds key facts for better chatbot replies [OK]

Common Mistakes:

• Thinking NER confuses chatbots
• Assuming NER translates messages
• Believing NER deletes input