Named Entity Recognition basics in ML Python - Model Metrics & Evaluation

Metrics & Evaluation - Named Entity Recognition basics

Which metric matters for Named Entity Recognition and WHY

Named Entity Recognition (NER) finds names like people, places, or dates in text. We want to know how well the model finds these names correctly.

Precision tells us how many found names are actually correct. High precision means few wrong names.

Recall tells us how many real names the model found out of all names. High recall means few missed names.

F1 score balances precision and recall. It is the best single number to see overall NER quality.

We use these because NER is about finding exact names, so both missing names and wrong names matter.

Confusion matrix for Named Entity Recognition

    |-----------------------------|
    |           | Predicted       |
    | Actual    | Entity | Not Entity |
    |-----------------------------|
    | Entity    |  TP    |    FN     |
    | Not Entity|  FP    |    TN     |
    |-----------------------------|

    TP = Correctly found entities
    FP = Wrongly found entities (false alarms)
    FN = Missed entities
    TN = Correctly ignored non-entities
    

Example: If model finds 80 correct names (TP), misses 20 names (FN), and wrongly tags 10 words as names (FP), precision = 80/(80+10)=0.89, recall = 80/(80+20)=0.8.

Precision vs Recall tradeoff with examples

Imagine a NER model for medical records:

• High precision, low recall: Model finds only very sure names, so few wrong names but misses many real ones. Good if wrong names cause big problems.
• High recall, low precision: Model finds almost all names but also many wrong ones. Good if missing a name is worse than extra false names.

Choosing depends on what matters more: avoiding false names or missing real names.

What good vs bad metric values look like for NER

• Good: Precision and recall both above 0.85, F1 score above 0.85 means model finds most names correctly and misses few.
• Bad: Precision or recall below 0.5 means many wrong names or many missed names, making the model unreliable.
• Very high precision but very low recall means model is too strict, missing many names.
• Very high recall but very low precision means model is too loose, tagging many wrong names.

Common pitfalls in NER metrics

• Accuracy paradox: Most words are not names, so a model that tags nothing can have high accuracy but is useless.
• Data leakage: If test data has same sentences as training, metrics look better but model may fail on new text.
• Overfitting: Very high training metrics but low test metrics means model memorizes training names, not generalizing well.
• Ignoring entity boundaries: Partial matches count as errors, so metrics must consider exact entity spans.

Self-check question

Your NER model has 98% accuracy but only 12% recall on entities. Is it good for production?

Answer: No. The high accuracy is misleading because most words are not entities. The very low recall means the model misses almost all real names, so it is not useful.

Key Result

F1 score balances precision and recall to best measure Named Entity Recognition quality.

Practice

1. What is the main goal of Named Entity Recognition (NER) in text processing?

easy

A. To generate new sentences based on input

B. To find and label important words like names and places

C. To count the number of words in a sentence

D. To translate text from one language to another

Solution

1. Step 1: Understand NER purpose

   NER is designed to identify and label entities such as names, places, and organizations in text.

2. Step 2: Compare options

   Only To find and label important words like names and places describes this goal correctly; others describe different tasks like translation or counting.

3. Final Answer:

   To find and label important words like names and places -> Option B

4. Quick Check:

   NER = Label important words [OK]

Hint: NER labels names and places in text quickly [OK]

Common Mistakes:

• Confusing NER with translation
• Thinking NER counts words
• Mixing NER with text generation

2. Which of the following is the correct way to label the entity 'London' as a location in a simple NER tagging scheme?

easy

A. London

B. LOC-London

C. [LOC]London[/LOC]

D. B-LOC

Solution

1. Step 1: Recall common NER tagging formats

   NER often uses tags like B-LOC (Beginning of Location) and I-LOC (Inside Location) in BIO format.

2. Step 2: Identify correct tag for 'London'

   Since 'London' is a single word entity, it is labeled as B-LOC to mark the start of a location entity.

3. Final Answer:

   B-LOC -> Option D

4. Quick Check:

   B-LOC tags start of location entity [OK]

Hint: Use B-LOC to mark start of location entity [OK]

Common Mistakes:

• Using XML-like tags which are not standard in NER
• Confusing bracket styles
• Writing label after the word instead of before

3. Given this simple NER output for the sentence: ['John', 'B-PER'], ['lives', 'O'], ['in', 'O'], ['Paris', 'B-LOC'], what is the recognized entity?

medium

A. No entities recognized

B. John is a location, Paris is a person

C. John is a person, Paris is a location

D. John and Paris are both organizations

Solution

1. Step 1: Interpret tags

   'B-PER' means beginning of a person entity, 'B-LOC' means beginning of a location entity, 'O' means outside any entity.

2. Step 2: Map tags to words

   John is tagged as a person, Paris as a location, and 'lives' and 'in' are not entities.

3. Final Answer:

   John is a person, Paris is a location -> Option C

4. Quick Check:

   B-PER = person, B-LOC = location [OK]

Hint: B-PER means person, B-LOC means location [OK]

Common Mistakes:

• Mixing up person and location tags
• Ignoring 'O' tags as no entity
• Assuming all words are entities

4. You have this NER tagging output: ['Apple', 'I-ORG'], ['is', 'O'], ['big', 'O']. What is the error in this tagging?

medium

A. 'Apple' should be tagged as B-ORG, not I-ORG

B. 'is' should be tagged as B-ORG

C. 'big' should be tagged as I-ORG

D. No error, tags are correct

Solution

1. Step 1: Understand BIO tagging rules

   'B-' means beginning of an entity, 'I-' means inside an entity. The first word of an entity must be B- tag.

2. Step 2: Check 'Apple' tag

   'Apple' is the first word of the organization entity, so it must be tagged B-ORG, not I-ORG.

3. Final Answer:

   'Apple' should be tagged as B-ORG, not I-ORG -> Option A

4. Quick Check:

   First entity word = B- tag [OK]

Hint: First entity word must use B- tag [OK]

Common Mistakes:

• Tagging first entity word as I-
• Tagging non-entity words as entities
• Ignoring BIO format rules

5. You want to improve a simple NER model trained on a small dataset. Which approach is best to get better results on recognizing entities in new text?

hard

A. Use a pretrained NER model and fine-tune it on your data

B. Increase the number of epochs without changing the model

C. Remove all entity labels and train a new model from scratch

D. Use random labels to increase data size

Solution

1. Step 1: Understand pretrained model benefits

   Pretrained NER models have learned from large data and generalize well to new text.

2. Step 2: Fine-tuning improves performance

   Fine-tuning a pretrained model on your small dataset adapts it to your specific task and improves accuracy.

3. Final Answer:

   Use a pretrained NER model and fine-tune it on your data -> Option A

4. Quick Check:

   Pretrained + fine-tune = best for small data [OK]

Hint: Fine-tune pretrained models for best NER results [OK]

Common Mistakes:

• Training from scratch on small data
• Ignoring pretrained models
• Using random labels that confuse training