NER with spaCy in NLP - Model Metrics & Evaluation

Metrics & Evaluation - NER with spaCy

Which metric matters for NER with spaCy and WHY

In Named Entity Recognition (NER), we want to find and label words like names, places, or dates correctly. The key metrics are Precision, Recall, and F1-score.

Precision tells us how many of the entities the model found are actually correct. This matters because we don't want to label wrong words as entities.

Recall tells us how many of the real entities the model found. This matters because missing important entities means the model is incomplete.

F1-score balances precision and recall. It gives one number to see how well the model does overall.

We use these metrics because NER is about both finding entities and labeling them correctly.

Confusion matrix for NER (simplified)

          | Predicted Entity | Predicted Non-Entity
    ------|------------------|---------------------
    True  |        TP        |          FN         
    Entity| (correct entity) | (missed entity)     
    ------|------------------|---------------------
    True  |        FP        |          TN         
    Non-  | (wrong entity)   | (correct non-entity) 
    Entity|                  |                     
    

TP = True Positives: entities correctly found.
FP = False Positives: wrong words labeled as entities.
FN = False Negatives: real entities missed.
TN = True Negatives: non-entities correctly ignored.

Precision vs Recall tradeoff with examples

If the model has high precision but low recall, it means it labels entities carefully but misses many real ones. For example, a medical NER system that only tags very sure disease names but misses rare diseases.

If the model has high recall but low precision, it finds most entities but also labels many wrong words. For example, a news NER system that tags many words as people but includes many mistakes.

For NER, a good balance (high F1-score) is important because we want to find most entities and be correct.

What good vs bad metric values look like for NER

Good NER model: Precision, Recall, and F1-score all above 85%. This means it finds most entities and labels them correctly.

Bad NER model: Precision or Recall below 50%. This means it either misses many entities or makes many wrong labels.

Example: Precision=90%, Recall=40% means many entities missed (bad recall). Precision=40%, Recall=90% means many wrong labels (bad precision).

Common pitfalls in NER metrics

• Ignoring entity boundaries: Partial matches are not counted as correct. The model must get the full entity right.
• Data leakage: Testing on data the model saw during training inflates metrics falsely.
• Imbalanced entities: Some entity types may be rare, so overall metrics can hide poor performance on rare types.
• Overfitting: Very high training scores but low test scores mean the model memorizes instead of learning.

Self-check: Your model has 98% accuracy but 12% recall on entities. Is it good?

No, this model is not good for NER. The high accuracy is misleading because most words are not entities, so the model guesses "non-entity" most of the time and is right.

The very low recall (12%) means it misses almost all real entities. This defeats the purpose of NER, which is to find entities.

Better metrics to trust are precision, recall, and F1-score on the entity class, not overall accuracy.

Key Result

For NER with spaCy, F1-score balancing precision and recall best shows model quality.

Practice

1. What does NER (Named Entity Recognition) do in natural language processing?

easy

A. It generates new text based on input prompts.

B. It translates text from one language to another.

C. It summarizes long documents into short paragraphs.

D. It finds and labels important names and terms in text automatically.

Solution

1. Step 1: Understand NER's purpose

   NER identifies specific names like people, places, or organizations in text.

2. Step 2: Compare with other NLP tasks

   Translation, summarization, and text generation are different tasks than NER.

3. Final Answer:

   It finds and labels important names and terms in text automatically. -> Option D

4. Quick Check:

   NER = Finds names and terms [OK]

Hint: NER extracts names and terms, not translations or summaries [OK]

Common Mistakes:

• Confusing NER with translation or summarization
• Thinking NER generates new text
• Believing NER only finds keywords, not named entities

2. Which of the following is the correct way to load a pre-trained spaCy model for NER?

easy

A. import spacy; nlp = spacy.load('en_core_web_sm')

B. import spacy; nlp = spacy.model('en_core_web_sm')

C. import spacy; nlp = spacy.load_model('en_core_web_sm')

D. import spacy; nlp = spacy.get('en_core_web_sm')

Solution

1. Step 1: Recall spaCy model loading syntax

   spaCy uses spacy.load('model_name') to load pre-trained models.

2. Step 2: Check each option

   Only import spacy; nlp = spacy.load('en_core_web_sm') uses spacy.load correctly; others use invalid functions.

3. Final Answer:

   import spacy; nlp = spacy.load('en_core_web_sm') -> Option A

4. Quick Check:

   spaCy model loading = spacy.load() [OK]

Hint: Use spacy.load('model_name') to load models [OK]

Common Mistakes:

• Using spacy.model or spacy.load_model which don't exist
• Trying spacy.get which is not a spaCy function
• Forgetting to import spacy before loading

3. Given this code snippet using spaCy for NER:

import spacy
nlp = spacy.load('en_core_web_sm')
doc = nlp('Apple is looking at buying U.K. startup for $1 billion')
entities = [(ent.text, ent.label_) for ent in doc.ents]
print(entities)

What will be the output?

medium

A. [('Apple', 'PERSON'), ('U.K.', 'ORG'), ('$1 billion', 'QUANTITY')]

B. [('Apple', 'ORG'), ('startup', 'ORG'), ('$1 billion', 'MONEY')]

C. [('Apple', 'ORG'), ('U.K.', 'GPE'), ('$1 billion', 'MONEY')]

D. [('Apple', 'GPE'), ('U.K.', 'GPE'), ('$1 billion', 'MONEY')]

Solution

1. Step 1: Understand spaCy NER labels

   Apple is recognized as an organization (ORG), U.K. as geopolitical entity (GPE), and $1 billion as money (MONEY).

2. Step 2: Match entities with labels

   [('Apple', 'ORG'), ('U.K.', 'GPE'), ('$1 billion', 'MONEY')] correctly matches these entities and labels as spaCy outputs.

3. Final Answer:

   [('Apple', 'ORG'), ('U.K.', 'GPE'), ('$1 billion', 'MONEY')] -> Option C

4. Quick Check:

   spaCy NER output matches [('Apple', 'ORG'), ('U.K.', 'GPE'), ('$1 billion', 'MONEY')] [OK]

Hint: Check spaCy's common entity labels for correct matches [OK]

Common Mistakes:

• Confusing ORG with PERSON or GPE
• Mislabeling MONEY as QUANTITY
• Including words like 'startup' as entities

4. You run this code but get an error:

import spacy
doc = nlp('Google is a tech giant')

What is the most likely cause?

medium

A. spaCy does not support the word 'Google'.

B. The variable 'nlp' is not defined before use.

C. The text input is too short for NER.

D. Missing parentheses in the print statement.

Solution

1. Step 1: Check variable definitions

   The code uses 'nlp' without defining it by loading a spaCy model first.

2. Step 2: Identify error cause

   This causes a NameError because 'nlp' is undefined.

3. Final Answer:

   The variable 'nlp' is not defined before use. -> Option B

4. Quick Check:

   Undefined variable 'nlp' causes error [OK]

Hint: Always load model with spacy.load before using nlp [OK]

Common Mistakes:

• Assuming text length causes error
• Thinking spaCy can't recognize common words
• Confusing print syntax errors with variable errors

5. You want to extract only person names from a text using spaCy's NER. Which code snippet correctly filters for persons?

hard

A. persons = [ent.text for ent in doc.ents if ent.label_ == 'PERSON']

B. persons = [ent.text for ent in doc.ents if ent.label_ == 'ORG']

C. persons = [ent.text for ent in doc.ents if ent.label_ == 'GPE']

D. persons = [ent.text for ent in doc.ents if ent.label_ == 'MONEY']

Solution

1. Step 1: Identify label for persons in spaCy

   spaCy uses 'PERSON' label for people names.

2. Step 2: Filter entities by 'PERSON'

   Filtering doc.ents by ent.label_ == 'PERSON' extracts only person names.

3. Final Answer:

   persons = [ent.text for ent in doc.ents if ent.label_ == 'PERSON'] -> Option A

4. Quick Check:

   Filter entities by 'PERSON' label [OK]

Hint: Filter entities with label_ == 'PERSON' to get names [OK]

Common Mistakes:

• Using wrong labels like ORG or GPE for persons
• Not filtering entities at all
• Confusing entity text with label