Custom NER training basics in NLP - Model Metrics & Evaluation

Metrics & Evaluation - Custom NER training basics

Which metric matters for Custom NER training and WHY

In custom Named Entity Recognition (NER), the key metrics are Precision, Recall, and F1-score. These metrics tell us how well the model finds the correct entities and avoids mistakes.

Precision shows how many of the entities the model found are actually correct. This matters because we want to trust the entities the model highlights.

Recall shows how many of the real entities the model found. This matters because missing important entities can cause problems.

F1-score balances precision and recall, giving a single number to understand overall quality.

Confusion matrix for NER (simplified)

    |---------------------------|
    |           | Predicted     |
    | Actual    | Entity | No Entity |
    |---------------------------|
    | Entity    |   TP   |    FN    |
    | No Entity |   FP   |    TN    |
    |---------------------------|

    TP = Correctly found entities
    FP = Wrongly found entities (false alarms)
    FN = Missed entities
    TN = Correctly ignored non-entities
    

Precision = TP / (TP + FP)

Recall = TP / (TP + FN)

F1 = 2 * (Precision * Recall) / (Precision + Recall)

Precision vs Recall tradeoff with examples

If your NER model has high precision but low recall, it means it finds entities very accurately but misses many. For example, a medical NER that only tags very obvious diseases but misses rare ones.

If your model has high recall but low precision, it finds most entities but also tags many wrong ones. For example, tagging many words as diseases, including normal words.

Depending on your use case, you might want to favor one. For legal documents, missing an entity (low recall) might be worse. For chatbots, wrong tags (low precision) might confuse users.

What good vs bad metric values look like for Custom NER

Good: Precision and recall both above 85%, F1-score above 85%. This means the model finds most entities correctly and misses few.

Bad: Precision or recall below 50%, F1-score below 60%. This means many wrong tags or many missed entities, making the model unreliable.

Example: Precision=90%, Recall=40% means many entities are missed (bad recall). Precision=40%, Recall=90% means many false tags (bad precision).

Common pitfalls in NER metrics

• Accuracy paradox: Accuracy can be misleading because most words are not entities. A model tagging no entities can have high accuracy but is useless.
• Data leakage: If training and test data share sentences, metrics look better but model won't generalize.
• Overfitting: Very high training metrics but low test metrics means model memorized training data, not learned general rules.
• Ignoring entity types: Treating all entities the same can hide poor performance on important entity types.

Self-check question

Your custom NER model has 98% accuracy but only 12% recall on the entity class. Is it good for production? Why or why not?

Answer: No, it is not good. The high accuracy is misleading because most words are not entities. The very low recall means the model misses almost all real entities, which defeats the purpose of NER.

Key Result

Precision, recall, and F1-score are key to evaluate custom NER; accuracy alone is misleading due to class imbalance.

Practice

1. What is the main goal of custom NER training in NLP?

easy

A. To summarize long documents automatically

B. To teach the model to recognize specific words or phrases you label

C. To translate text from one language to another

D. To generate new text based on a prompt

Solution

1. Step 1: Understand what NER means

   NER stands for Named Entity Recognition, which means finding specific words or phrases in text.

2. Step 2: Identify the purpose of custom training

   Custom NER training teaches the model to find your special labeled words, not general tasks like translation or summarization.

3. Final Answer:

   To teach the model to recognize specific words or phrases you label -> Option B

4. Quick Check:

   Custom NER = Recognize labeled words [OK]

Hint: Custom NER means teaching model your special words [OK]

Common Mistakes:

• Confusing NER with translation or summarization
• Thinking NER generates new text
• Assuming NER works without labeled data

2. Which of the following is the correct way to label a sentence for custom NER training in Python spaCy format?

easy

A. ('Apple is a company', {'entities': [(0, 5, 'ORG')]})

B. ('Apple is a company', {'labels': [(0, 5, 'ORG')]})

C. ('Apple is a company', {'entities': [(6, 7, 'ORG')]})

D. ('Apple is a company', {'entities': [(0, 5, 'PERSON')]})

Solution

1. Step 1: Check the labeling key

   spaCy uses the 'entities' key, not 'labels', to hold labeled spans.

2. Step 2: Verify the span and label

   Span (0,5) covers 'Apple' correctly, and label 'ORG' (organization) fits. A span like (6,7,'ORG') points to the wrong position, and 'PERSON' is incorrect for a company.

3. Final Answer:

   ('Apple is a company', {'entities': [(0, 5, 'ORG')]}) -> Option A

4. Quick Check:

   Correct key and span = ('Apple is a company', {'entities': [(0, 5, 'ORG')]}) [OK]

Hint: Use 'entities' key with correct span and label [OK]

Common Mistakes:

• Using 'labels' instead of 'entities'
• Incorrect character span for entity
• Wrong entity type label

3. Given this training data snippet for custom NER:

TRAIN_DATA = [
  ('I love Paris', {'entities': [(7, 12, 'GPE')]})
]

What will the model predict for the sentence 'I love Paris' after training?

medium

A. [] (no entities)

B. [('I', 'GPE')]

C. [('Paris', 'GPE')]

D. [('love', 'GPE')]

Solution

1. Step 1: Understand the labeled entity

   The training data labels 'Paris' from character 7 to 12 as 'GPE' (Geopolitical entity).

2. Step 2: Predict model output after training

   The model learns to recognize 'Paris' as 'GPE' and should predict [('Paris', 'GPE')] for the same sentence.

3. Final Answer:

   [('Paris', 'GPE')] -> Option C

4. Quick Check:

   Entity span matches 'Paris' = [('Paris', 'GPE')] [OK]

Hint: Model predicts labeled spans from training data [OK]

Common Mistakes:

• Confusing entity span with other words
• Expecting no entities if training is done
• Mixing entity labels

4. You wrote this code to add a new entity label to your NER model:

ner.add_label('ANIMAL')

But after training, the model never detects 'ANIMAL' entities. What is the most likely mistake?

medium

A. The label 'ANIMAL' is reserved and cannot be used

B. You used the wrong method name; it should be add_entity()

C. You need to call ner.remove_label('ANIMAL') before adding

D. You forgot to include training examples with 'ANIMAL' labels

Solution

1. Step 1: Check the method usage

   ner.add_label('ANIMAL') is correct to add a new label. There is no add_entity() method, no need to call remove_label first, and 'ANIMAL' is not reserved.

2. Step 2: Verify training data

   Model learns from examples. Without training examples labeled 'ANIMAL', model cannot detect it.

3. Final Answer:

   You forgot to include training examples with 'ANIMAL' labels -> Option D

4. Quick Check:

   Training data needed for new labels = You forgot to include training examples with 'ANIMAL' labels [OK]

Hint: Add labeled examples for new entity labels [OK]

Common Mistakes:

• Assuming adding label alone trains model
• Using wrong method names
• Thinking labels are reserved keywords

5. You want to train a custom NER model to recognize two new entity types: 'FOOD' and 'DRINK'. You have labeled training data for both. Which of the following is the best approach to ensure the model learns both correctly?

hard

A. Add both labels with ner.add_label(), include balanced training examples for each, and train in multiple iterations

B. Add only 'FOOD' label first, train fully, then add 'DRINK' label and train again

C. Train the model without adding labels explicitly; it will learn automatically

D. Add labels but use only examples for 'FOOD' to avoid confusion

Solution

1. Step 1: Add all new labels before training

   Adding both 'FOOD' and 'DRINK' labels upfront ensures model knows what to learn.

2. Step 2: Provide balanced training data and train iteratively

   Balanced examples for both labels and multiple training loops help model learn both well.

3. Final Answer:

   Add both labels with ner.add_label(), include balanced training examples for each, and train in multiple iterations -> Option A

4. Quick Check:

   All labels + balanced data + training = Add both labels with ner.add_label(), include balanced training examples for each, and train in multiple iterations [OK]

Hint: Add all labels and balanced data before training [OK]

Common Mistakes:

• Adding labels one by one with separate training
• Skipping label addition
• Training with unbalanced or missing examples