Experiment - Named entity recognition

Problem:We want to teach a computer to find names of people, places, and organizations in sentences. Our current model learns well on training data but does not do well on new sentences.

Current Metrics:Training accuracy: 95%, Validation accuracy: 70%

Issue:The model is overfitting: it performs very well on training data but poorly on validation data.

Your Task

Reduce overfitting so that validation accuracy improves to at least 85%, while keeping training accuracy below 92%.

You can only change the model architecture and training settings.

Do not change the dataset or add more data.

Hint 1

Hint 2

Hint 3

Solution

NLP

import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Embedding, Bidirectional, LSTM, Dense, Dropout
from tensorflow.keras.callbacks import EarlyStopping

# Sample data placeholders (replace with actual data loading)
X_train = ...  # tokenized and padded sequences for training
y_train = ...  # one-hot encoded labels for training
X_val = ...    # tokenized and padded sequences for validation
y_val = ...    # one-hot encoded labels for validation

vocab_size = 10000  # example vocabulary size
embedding_dim = 64
max_len = 100  # max sequence length
num_classes = 10  # number of entity classes including 'O'

model = Sequential([
    Embedding(vocab_size, embedding_dim, input_length=max_len),
    Bidirectional(LSTM(64, return_sequences=True)),
    Dropout(0.5),
    Bidirectional(LSTM(32, return_sequences=True)),
    Dropout(0.5),
    Dense(num_classes, activation='softmax')
])

model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

early_stop = EarlyStopping(monitor='val_loss', patience=3, restore_best_weights=True)

history = model.fit(
    X_train, y_train,
    epochs=20,
    batch_size=32,
    validation_data=(X_val, y_val),
    callbacks=[early_stop]
)

# After training, evaluate on validation set
val_loss, val_accuracy = model.evaluate(X_val, y_val)
print(f'Validation accuracy: {val_accuracy * 100:.2f}%')

Added Dropout layers after LSTM layers to reduce overfitting.

Reduced the number of units in the second LSTM layer from 64 to 32 to lower model complexity.

Added EarlyStopping callback to stop training when validation loss stops improving.

Results Interpretation

Before: Training accuracy: 95%, Validation accuracy: 70% (overfitting)

After: Training accuracy: 90%, Validation accuracy: 87% (better generalization)

Adding dropout and early stopping helps the model generalize better by preventing it from memorizing training data, which reduces overfitting and improves validation accuracy.

Bonus Experiment

Try using a pre-trained language model like BERT for named entity recognition to see if it improves accuracy further.

💡 Hint

Use a library like Hugging Face Transformers to load a pre-trained BERT model and fine-tune it on your NER dataset.