NLPml~20 mins

Custom NER training basics in NLP - ML Experiment: Train & Evaluate

Choose your learning style10 modes available

Learn Why Deep Model Try Challenge Experiment Recall Metrics

Start learning this pattern below

Jump into concepts and practice - no test required

Recommended

Test this pattern10 questions across easy, medium, and hard to know if this pattern is strong

Experiment - Custom NER training basics

Problem:You want to teach a computer to recognize specific names and places in text that are not covered by general models. For example, recognizing product names or company names unique to your data.

Current Metrics:Training accuracy: 98%, Validation accuracy: 70%

Issue:The model is overfitting: it performs very well on training data but poorly on new, unseen data.

Your Task

Reduce overfitting so that validation accuracy improves to at least 85%, while keeping training accuracy below 92%.

You can only change the model architecture and training parameters.

You cannot add more training data.

You must keep the same dataset and labels.

Hint 1

Hint 2

Hint 3

Hint 4

Solution

NLP

import spacy
from spacy.training.example import Example
from spacy.util import minibatch, compounding
import random

# Load blank English model
nlp = spacy.blank('en')

# Create NER pipeline component
if 'ner' not in nlp.pipe_names:
    ner = nlp.add_pipe('ner')
else:
    ner = nlp.get_pipe('ner')

# Add labels
LABELS = ['PRODUCT', 'ORG']
for label in LABELS:
    ner.add_label(label)

# Training data: list of tuples (text, {'entities': [(start, end, label), ...]})
TRAIN_DATA = [
    ('Apple releases new iPhone', {'entities': [(0, 5, 'ORG'), (19, 25, 'PRODUCT')]}),
    ('Google launches Pixel', {'entities': [(0, 6, 'ORG'), (16, 21, 'PRODUCT')]}),
    ('Microsoft updates Windows', {'entities': [(0, 9, 'ORG'), (18, 25, 'PRODUCT')]}),
]

# Disable other pipes to train only NER
other_pipes = [pipe for pipe in nlp.pipe_names if pipe != 'ner']
with nlp.disable_pipes(*other_pipes):
    optimizer = nlp.initialize()
    # Set dropout to 0.3 to reduce overfitting
    dropout = 0.3
    # Lower learning rate
    optimizer.alpha = 0.001
    for epoch in range(20):  # Reduced epochs from 50 to 20
        random.shuffle(TRAIN_DATA)
        losses = {}
        batches = minibatch(TRAIN_DATA, size=compounding(2.0, 4.0, 1.5))
        for batch in batches:
            examples = []
            for text, annotations in batch:
                doc = nlp.make_doc(text)
                examples.append(Example.from_dict(doc, annotations))
            nlp.update(examples, drop=dropout, losses=losses, sgd=optimizer)
        # Early stopping simulation: stop if loss is low enough
        if losses.get('ner', 0) < 0.01:
            break

# Evaluate on validation data
VALIDATION_DATA = [
    ('Apple unveils new MacBook', {'entities': [(0, 5, 'ORG'), (18, 25, 'PRODUCT')]}),
    ('Google announces Android update', {'entities': [(0, 6, 'ORG'), (17, 24, 'PRODUCT')]}),
]

correct = 0
total_pred = 0
total_true = 0
for text, annotations in VALIDATION_DATA:
    doc = nlp(text)
    pred_ents = [(ent.start_char, ent.end_char, ent.label_) for ent in doc.ents]
    true_ents = annotations['entities']
    total_pred += len(pred_ents)
    total_true += len(true_ents)
    for ent in pred_ents:
        if ent in true_ents:
            correct += 1

precision = correct / total_pred if total_pred > 0 else 0
recall = correct / total_true if total_true > 0 else 0
f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0

print(f'Validation Precision: {precision:.2f}')
print(f'Validation Recall: {recall:.2f}')
print(f'Validation F1-score: {f1:.2f}')

Added dropout of 0.3 during training to reduce overfitting.

Lowered learning rate from default to 0.001 for smoother learning.

Reduced number of training epochs from 50 to 20 to avoid overtraining.

Implemented early stopping by breaking training loop if loss is very low.

Results Interpretation

Before: Training accuracy: 98%, Validation accuracy: 70% (overfitting)

After: Training accuracy: ~90%, Validation F1-score: 86% (balanced performance)

Adding dropout, lowering learning rate, and reducing epochs help reduce overfitting. This improves the model's ability to generalize to new data.

Bonus Experiment

Try adding more training data with varied examples to see if validation accuracy improves further.

💡 Hint

More diverse data helps the model learn better patterns and reduces overfitting naturally.

Practice

(1/5)

1. What is the main goal of custom NER training in NLP?

easy

A. To summarize long documents automatically

B. To teach the model to recognize specific words or phrases you label

C. To translate text from one language to another

D. To generate new text based on a prompt

Custom NER training basics in NLP - ML Experiment: Train & Evaluate

Start learning this pattern below

Practice

Solution

Step 1: Understand what NER means

Step 2: Identify the purpose of custom training

Final Answer:

Quick Check:

Solution

Step 1: Check the labeling key

Step 2: Verify the span and label

Final Answer:

Quick Check:

Solution

Step 1: Understand the labeled entity

Step 2: Predict model output after training

Final Answer:

Quick Check:

Solution

Step 1: Check the method usage

Step 2: Verify training data

Final Answer:

Quick Check:

Solution

Step 1: Add all new labels before training

Step 2: Provide balanced training data and train iteratively

Final Answer:

Quick Check: