Named Entity Recognition basics in ML Python

Introduction

Named Entity Recognition helps computers find and label important things like names, places, and dates in text. This makes it easier to understand and organize information.

When you want to find all the people mentioned in a news article.

When you need to extract company names from customer reviews.

When you want to identify locations in travel blogs automatically.

When you want to organize emails by detecting dates and events.

When you want to help a chatbot understand user questions better by recognizing key terms.

Syntax

ML Python

model = SomeNERModel()
predictions = model.predict(text)

NER models take text as input and output labels for each word or phrase.

Labels usually include categories like PERSON, LOCATION, ORGANIZATION, and DATE.

Examples

The model finds names, places, and dates in the sentence.

ML Python

text = "Alice went to Paris in April."
predictions = model.predict(text)
# Output: [('Alice', 'PERSON'), ('Paris', 'LOCATION'), ('April', 'DATE')]

The model detects company and person names.

ML Python

text = "Google was founded by Larry Page and Sergey Brin."
predictions = model.predict(text)
# Output: [('Google', 'ORGANIZATION'), ('Larry Page', 'PERSON'), ('Sergey Brin', 'PERSON')]

Sample Model

This example trains a simple Named Entity Recognition model using a CRF on a tiny dataset. It then predicts entity labels for a new sentence.

ML Python

from sklearn_crfsuite import CRF

# Sample training data: words and their entity labels
train_sents = [[('John', 'B-PER'), ('lives', 'O'), ('in', 'O'), ('New', 'B-LOC'), ('York', 'I-LOC')]]

# Feature extractor for each word
def word2features(sent, i):
    word = sent[i][0]
    features = {
        'word.lower()': word.lower(),
        'word.isupper()': word.isupper(),
        'word.istitle()': word.istitle(),
        'word.isdigit()': word.isdigit(),
    }
    return features

def sent2features(sent):
    return [word2features(sent, i) for i in range(len(sent))]

def sent2labels(sent):
    return [label for token, label in sent]

X_train = [sent2features(s) for s in train_sents]
y_train = [sent2labels(s) for s in train_sents]

# Train CRF model
crf = CRF(algorithm='lbfgs', max_iterations=100)
crf.fit(X_train, y_train)

# Test sentence
test_sent = [('Mary', 'O'), ('moved', 'O'), ('to', 'O'), ('Los', 'O'), ('Angeles', 'O')]
X_test = [word2features(test_sent, i) for i in range(len(test_sent))]

# Predict entity labels
predicted = crf.predict([X_test])[0]

# Print results
for (word, _), label in zip(test_sent, predicted):
    print(f"{word}: {label}")

Important Notes

NER models often use labels like B- (beginning), I- (inside), and O (outside) to mark entity spans.

Real NER models need much more data to work well.

Pretrained models like spaCy or Hugging Face transformers can do NER without training.

Summary

Named Entity Recognition finds and labels important words like names and places in text.

NER uses special labels to mark where entities start and continue.

Simple models can be trained on small data, but pretrained models work best for real tasks.

Practice

1. What is the main goal of Named Entity Recognition (NER) in text processing?

easy

A. To generate new sentences based on input

B. To find and label important words like names and places

C. To count the number of words in a sentence

D. To translate text from one language to another

Solution

1. Step 1: Understand NER purpose

   NER is designed to identify and label entities such as names, places, and organizations in text.

2. Step 2: Compare options

   Only To find and label important words like names and places describes this goal correctly; others describe different tasks like translation or counting.

3. Final Answer:

   To find and label important words like names and places -> Option B

4. Quick Check:

   NER = Label important words [OK]

Hint: NER labels names and places in text quickly [OK]

Common Mistakes:

• Confusing NER with translation
• Thinking NER counts words
• Mixing NER with text generation

2. Which of the following is the correct way to label the entity 'London' as a location in a simple NER tagging scheme?

easy

A. London

B. LOC-London

C. [LOC]London[/LOC]

D. B-LOC

Solution

1. Step 1: Recall common NER tagging formats

   NER often uses tags like B-LOC (Beginning of Location) and I-LOC (Inside Location) in BIO format.

2. Step 2: Identify correct tag for 'London'

   Since 'London' is a single word entity, it is labeled as B-LOC to mark the start of a location entity.

3. Final Answer:

   B-LOC -> Option D

4. Quick Check:

   B-LOC tags start of location entity [OK]

Hint: Use B-LOC to mark start of location entity [OK]

Common Mistakes:

• Using XML-like tags which are not standard in NER
• Confusing bracket styles
• Writing label after the word instead of before

3. Given this simple NER output for the sentence: ['John', 'B-PER'], ['lives', 'O'], ['in', 'O'], ['Paris', 'B-LOC'], what is the recognized entity?

medium

A. No entities recognized

B. John is a location, Paris is a person

C. John is a person, Paris is a location

D. John and Paris are both organizations

Solution

1. Step 1: Interpret tags

   'B-PER' means beginning of a person entity, 'B-LOC' means beginning of a location entity, 'O' means outside any entity.

2. Step 2: Map tags to words

   John is tagged as a person, Paris as a location, and 'lives' and 'in' are not entities.

3. Final Answer:

   John is a person, Paris is a location -> Option C

4. Quick Check:

   B-PER = person, B-LOC = location [OK]

Hint: B-PER means person, B-LOC means location [OK]

Common Mistakes:

• Mixing up person and location tags
• Ignoring 'O' tags as no entity
• Assuming all words are entities

4. You have this NER tagging output: ['Apple', 'I-ORG'], ['is', 'O'], ['big', 'O']. What is the error in this tagging?

medium

A. 'Apple' should be tagged as B-ORG, not I-ORG

B. 'is' should be tagged as B-ORG

C. 'big' should be tagged as I-ORG

D. No error, tags are correct

Solution

1. Step 1: Understand BIO tagging rules

   'B-' means beginning of an entity, 'I-' means inside an entity. The first word of an entity must be B- tag.

2. Step 2: Check 'Apple' tag

   'Apple' is the first word of the organization entity, so it must be tagged B-ORG, not I-ORG.

3. Final Answer:

   'Apple' should be tagged as B-ORG, not I-ORG -> Option A

4. Quick Check:

   First entity word = B- tag [OK]

Hint: First entity word must use B- tag [OK]

Common Mistakes:

• Tagging first entity word as I-
• Tagging non-entity words as entities
• Ignoring BIO format rules

5. You want to improve a simple NER model trained on a small dataset. Which approach is best to get better results on recognizing entities in new text?

hard

A. Use a pretrained NER model and fine-tune it on your data

B. Increase the number of epochs without changing the model

C. Remove all entity labels and train a new model from scratch

D. Use random labels to increase data size

Solution

1. Step 1: Understand pretrained model benefits

   Pretrained NER models have learned from large data and generalize well to new text.

2. Step 2: Fine-tuning improves performance

   Fine-tuning a pretrained model on your small dataset adapts it to your specific task and improves accuracy.

3. Final Answer:

   Use a pretrained NER model and fine-tune it on your data -> Option A

4. Quick Check:

   Pretrained + fine-tune = best for small data [OK]

Hint: Fine-tune pretrained models for best NER results [OK]

Common Mistakes:

• Training from scratch on small data
• Ignoring pretrained models
• Using random labels that confuse training