spaCy installation and models in NLP - Model Metrics & Evaluation

Metrics & Evaluation - spaCy installation and models

Which metric matters for spaCy models and WHY

When using spaCy models for tasks like text classification or named entity recognition, the key metrics to watch are Precision, Recall, and F1-score. These metrics tell us how well the model finds the right information in text.

Precision shows how many of the model's positive predictions are actually correct. Recall shows how many of the actual positive cases the model found. F1-score balances both precision and recall into one number.

We focus on these because spaCy models often work with unbalanced data, like rare entities or categories, where accuracy alone can be misleading.

Confusion matrix example for spaCy text classification

          Predicted Positive   Predicted Negative
Actual Positive       80                 20
Actual Negative       10                 90

Total samples = 200

Precision = 80 / (80 + 10) = 0.89
Recall = 80 / (80 + 20) = 0.80
F1-score = 2 * (0.89 * 0.80) / (0.89 + 0.80) = 0.84

This matrix helps us see where the model makes mistakes: false positives (10) and false negatives (20).

Precision vs Recall tradeoff with spaCy models

Imagine spaCy is used to find names of medicines in text. If the model has high precision, it means most found names are correct, but it might miss some names (lower recall).

If it has high recall, it finds almost all medicine names but might include wrong words (lower precision).

Choosing which to prioritize depends on the task. For example, in medical text, missing a medicine name (low recall) can be worse than having some extra wrong names (lower precision).

What good vs bad metric values look like for spaCy models

• Good: Precision and recall both above 0.85, F1-score close to 0.85 or higher. This means the model finds most correct items and makes few mistakes.
• Bad: Precision or recall below 0.5 means the model misses many correct items or makes many wrong predictions. F1-score below 0.6 usually shows poor balance.

Common pitfalls in spaCy model metrics

• Accuracy paradox: High accuracy can happen if one class dominates, but the model fails on rare classes.
• Data leakage: If test data leaks into training, metrics look too good but model fails in real use.
• Overfitting: Very high training metrics but low test metrics mean the model memorizes training data and won't generalize.

Self-check question

Your spaCy model for entity recognition has 98% accuracy but only 12% recall on rare entities. Is it good for production? Why or why not?

Answer: No, it is not good. The low recall means the model misses most rare entities, which can be critical depending on the task. High accuracy is misleading here because most data is not rare entities.

Key Result

Precision, recall, and F1-score are key to evaluate spaCy models, especially for unbalanced NLP tasks.

Practice

1. What is the correct command to install spaCy using pip?

easy

A. pip install spacy-model

B. pip install spacy

C. python -m spacy install

D. pip download spacy

Solution

1. Step 1: Understand pip installation command

   The standard way to install Python packages is using pip install package_name.

2. Step 2: Identify spaCy package name

   The correct package name for spaCy is exactly spacy, so the command is pip install spacy.

3. Final Answer:

   pip install spacy -> Option B

4. Quick Check:

   pip install spacy = B [OK]

Hint: Use 'pip install spacy' to install the main package [OK]

Common Mistakes:

• Using 'pip install spacy-model' which is incorrect
• Trying 'python -m spacy install' which is invalid
• Using 'pip download spacy' which only downloads, not installs

2. Which command correctly downloads the English small model for spaCy?

easy

A. python -m spacy download en_core_web_sm

B. pip install en_core_web_sm

C. spacy install en_core_web_sm

D. python spacy download en

Solution

1. Step 1: Identify the correct download command format

   spaCy models are downloaded using python -m spacy download <model_name>.

2. Step 2: Match the correct model name for English small

   The official English small model is named en_core_web_sm, so the full command is python -m spacy download en_core_web_sm.

3. Final Answer:

   python -m spacy download en_core_web_sm -> Option A

4. Quick Check:

   python -m spacy download en_core_web_sm = A [OK]

Hint: Use 'python -m spacy download model_name' to get models [OK]

Common Mistakes:

• Trying to install models with pip instead of spacy download
• Using incomplete model names like 'en' only
• Using 'spacy install' which is not a valid command

3. What will be the output of this code snippet after loading the spaCy English model and processing text?

import spacy
nlp = spacy.load('en_core_web_sm')
doc = nlp('Hello world!')
print([(token.text, token.pos_) for token in doc])

medium

A. [('Hello', 'NOUN'), ('world', 'VERB'), ('!', 'PUNCT')]

B. [('Hello', 'VERB'), ('world', 'ADJ'), ('!', 'SYM')]

C. [('Hello', 'INTJ'), ('world', 'NOUN'), ('!', 'PUNCT')]

D. SyntaxError

Solution

1. Step 1: Load the English model and process text

   The code loads the 'en_core_web_sm' model and processes the sentence 'Hello world!'.

2. Step 2: Understand token parts of speech

   In spaCy, 'Hello' is tagged as interjection (INTJ), 'world' as noun (NOUN), and '!' as punctuation (PUNCT).

3. Final Answer:

   [('Hello', 'INTJ'), ('world', 'NOUN'), ('!', 'PUNCT')] -> Option C

4. Quick Check:

   Token POS tags match [('Hello', 'INTJ'), ('world', 'NOUN'), ('!', 'PUNCT')] [OK]

Hint: Remember 'Hello' is INTJ, 'world' is NOUN, '!' is PUNCT [OK]

Common Mistakes:

• Confusing POS tags like 'Hello' as VERB
• Expecting syntax error due to correct code
• Mixing up punctuation tag as SYM instead of PUNCT

4. Identify the error in this code snippet for loading a spaCy model:

import spacy
nlp = spacy.load('en_core_web_md')
doc = nlp('Test sentence.')
print(doc.text)

Assuming the model was not downloaded yet.

medium

A. ModelNotFoundError because 'en_core_web_md' is not installed

B. SyntaxError due to wrong import

C. AttributeError on 'nlp' object

D. No error, code runs fine

Solution

1. Step 1: Check if model is downloaded

   The code tries to load 'en_core_web_md' model, which must be downloaded first using spaCy's download command.

2. Step 2: Understand error when model missing

   If the model is not installed, spaCy raises a ModelNotFoundError when calling spacy.load().

3. Final Answer:

   ModelNotFoundError because 'en_core_web_md' is not installed -> Option A

4. Quick Check:

   Missing model causes ModelNotFoundError [OK]

Hint: Always download models before loading to avoid ModelNotFoundError [OK]

Common Mistakes:

• Assuming code runs without downloading model
• Thinking import causes SyntaxError
• Expecting AttributeError on nlp object

5. You want to process text in French using spaCy. Which steps correctly install and load the French model for use in your Python code?

hard

A. Run python -m spacy install fr_core_news_sm then nlp = spacy.load('fr_core_news_sm')

B. Run pip install fr_core_news_sm then nlp = spacy.load('fr')

C. Run python -m spacy download fr then nlp = spacy.load('fr_core_news_sm')

D. Run python -m spacy download fr_core_news_sm then nlp = spacy.load('fr_core_news_sm')

Solution

1. Step 1: Download the correct French model

   The official French small model is named fr_core_news_sm, downloaded with python -m spacy download fr_core_news_sm.

2. Step 2: Load the model in Python

   After downloading, load it with spacy.load('fr_core_news_sm') to process French text.

3. Final Answer:

   Run python -m spacy download fr_core_news_sm then nlp = spacy.load('fr_core_news_sm') -> Option D

4. Quick Check:

   Download full model name, then load same name [OK]

Hint: Download full model name, then load it exactly [OK]

Common Mistakes:

• Trying to install model with pip instead of spacy download
• Using short name 'fr' in download command
• Using 'spacy install' which is invalid