Custom NER training basics in NLP - Model Metrics & Evaluation

Metrics & Evaluation - Custom NER training basics

Which metric matters for Custom NER training and WHY

In custom Named Entity Recognition (NER), the key metrics are Precision, Recall, and F1-score. These metrics tell us how well the model finds the correct entities and avoids mistakes.

Precision shows how many of the entities the model found are actually correct. This matters because we want to trust the entities the model highlights.

Recall shows how many of the real entities the model found. This matters because missing important entities can cause problems.

F1-score balances precision and recall, giving a single number to understand overall quality.

Confusion matrix for NER (simplified)

    |---------------------------|
    |           | Predicted     |
    | Actual    | Entity | No Entity |
    |---------------------------|
    | Entity    |   TP   |    FN    |
    | No Entity |   FP   |    TN    |
    |---------------------------|

    TP = Correctly found entities
    FP = Wrongly found entities (false alarms)
    FN = Missed entities
    TN = Correctly ignored non-entities
    

Precision = TP / (TP + FP)

Recall = TP / (TP + FN)

F1 = 2 * (Precision * Recall) / (Precision + Recall)

Precision vs Recall tradeoff with examples

If your NER model has high precision but low recall, it means it finds entities very accurately but misses many. For example, a medical NER that only tags very obvious diseases but misses rare ones.

If your model has high recall but low precision, it finds most entities but also tags many wrong ones. For example, tagging many words as diseases, including normal words.

Depending on your use case, you might want to favor one. For legal documents, missing an entity (low recall) might be worse. For chatbots, wrong tags (low precision) might confuse users.

What good vs bad metric values look like for Custom NER

Good: Precision and recall both above 85%, F1-score above 85%. This means the model finds most entities correctly and misses few.

Bad: Precision or recall below 50%, F1-score below 60%. This means many wrong tags or many missed entities, making the model unreliable.

Example: Precision=90%, Recall=40% means many entities are missed (bad recall). Precision=40%, Recall=90% means many false tags (bad precision).

Common pitfalls in NER metrics

• Accuracy paradox: Accuracy can be misleading because most words are not entities. A model tagging no entities can have high accuracy but is useless.
• Data leakage: If training and test data share sentences, metrics look better but model won't generalize.
• Overfitting: Very high training metrics but low test metrics means model memorized training data, not learned general rules.
• Ignoring entity types: Treating all entities the same can hide poor performance on important entity types.

Self-check question

Your custom NER model has 98% accuracy but only 12% recall on the entity class. Is it good for production? Why or why not?

Answer: No, it is not good. The high accuracy is misleading because most words are not entities. The very low recall means the model misses almost all real entities, which defeats the purpose of NER.

Key Result

Precision, recall, and F1-score are key to evaluate custom NER; accuracy alone is misleading due to class imbalance.