NER with NLTK in NLP - Model Metrics & Evaluation

Metrics & Evaluation - NER with NLTK

Which metric matters for NER with NLTK and WHY

Named Entity Recognition (NER) finds names like people, places, or organizations in text. The main metrics to check are Precision, Recall, and F1-score.

Precision tells us how many of the entities the model found are actually correct. This matters because we want to avoid wrong names.

Recall tells us how many of the real entities the model found. This matters because missing important names can be bad.

F1-score balances precision and recall to give one clear number showing overall quality.

Confusion matrix for NER (simplified)

          | Predicted Entity | Predicted Non-Entity
    ------|------------------|--------------------
    True  |        TP        |         FN
    Entity|                  |                    
    ------|------------------|--------------------
    True  |        FP        |         TN
    Non-  |                  |                    
    Entity|                  |                    

    TP = Correctly found entities
    FP = Wrongly found entities (false alarms)
    FN = Missed entities
    TN = Correctly ignored non-entities
    

Precision vs Recall tradeoff in NER

If you want to be very sure about the entities you find, focus on high precision. For example, a legal document analyzer should not mark wrong names.

If you want to find as many entities as possible, even if some are wrong, focus on high recall. For example, a news aggregator might want to catch all possible names.

Usually, improving one lowers the other. The F1-score helps find a good balance.

Good vs Bad metric values for NER

Good: Precision and recall above 0.8 means the model finds most names correctly and misses few.

Bad: Precision below 0.5 means many wrong names are found. Recall below 0.5 means many names are missed.

F1-score below 0.6 usually means the model needs improvement.

Common pitfalls in NER metrics

• Accuracy paradox: Most words are not entities, so accuracy can be high even if the model never finds entities.
• Data leakage: Testing on data the model saw during training inflates metrics falsely.
• Overfitting: Very high training scores but low test scores mean the model memorizes instead of learning.
• Ignoring entity types: Treating all entities the same can hide poor performance on important types.

Self-check question

Your NER model has 98% accuracy but only 12% recall on person names. Is it good for production?

Answer: No. The high accuracy is misleading because most words are not person names. The very low recall means the model misses almost all person names, which is bad if you need to find them.

Key Result

For NER with NLTK, F1-score balancing precision and recall best shows model quality.