spaCy installation and models in NLP - Model Metrics & Evaluation

Metrics & Evaluation - spaCy installation and models

Which metric matters for spaCy models and WHY

When using spaCy models for tasks like text classification or named entity recognition, the key metrics to watch are Precision, Recall, and F1-score. These metrics tell us how well the model finds the right information in text.

Precision shows how many of the model's positive predictions are actually correct. Recall shows how many of the actual positive cases the model found. F1-score balances both precision and recall into one number.

We focus on these because spaCy models often work with unbalanced data, like rare entities or categories, where accuracy alone can be misleading.

Confusion matrix example for spaCy text classification

          Predicted Positive   Predicted Negative
Actual Positive       80                 20
Actual Negative       10                 90

Total samples = 200

Precision = 80 / (80 + 10) = 0.89
Recall = 80 / (80 + 20) = 0.80
F1-score = 2 * (0.89 * 0.80) / (0.89 + 0.80) = 0.84

This matrix helps us see where the model makes mistakes: false positives (10) and false negatives (20).

Precision vs Recall tradeoff with spaCy models

Imagine spaCy is used to find names of medicines in text. If the model has high precision, it means most found names are correct, but it might miss some names (lower recall).

If it has high recall, it finds almost all medicine names but might include wrong words (lower precision).

Choosing which to prioritize depends on the task. For example, in medical text, missing a medicine name (low recall) can be worse than having some extra wrong names (lower precision).

What good vs bad metric values look like for spaCy models

• Good: Precision and recall both above 0.85, F1-score close to 0.85 or higher. This means the model finds most correct items and makes few mistakes.
• Bad: Precision or recall below 0.5 means the model misses many correct items or makes many wrong predictions. F1-score below 0.6 usually shows poor balance.

Common pitfalls in spaCy model metrics

• Accuracy paradox: High accuracy can happen if one class dominates, but the model fails on rare classes.
• Data leakage: If test data leaks into training, metrics look too good but model fails in real use.
• Overfitting: Very high training metrics but low test metrics mean the model memorizes training data and won't generalize.

Self-check question

Your spaCy model for entity recognition has 98% accuracy but only 12% recall on rare entities. Is it good for production? Why or why not?

Answer: No, it is not good. The low recall means the model misses most rare entities, which can be critical depending on the task. High accuracy is misleading here because most data is not rare entities.

Key Result

Precision, recall, and F1-score are key to evaluate spaCy models, especially for unbalanced NLP tasks.