Entity types (PERSON, ORG, LOC, DATE) in NLP - ML Experiment: Train & Evaluate

Experiment - Entity types (PERSON, ORG, LOC, DATE)

Problem:You want to build a model that can recognize named entities in text, specifically people (PERSON), organizations (ORG), locations (LOC), and dates (DATE). The current model identifies entities but often confuses entity types or misses some entities.

Current Metrics:Training accuracy: 92%, Validation accuracy: 75%, Validation F1-score: 0.70

Issue:The model is overfitting: training accuracy is high but validation accuracy and F1-score are much lower, indicating poor generalization.

Your Task

Reduce overfitting to improve validation accuracy to at least 85% and validation F1-score to at least 0.80, while keeping training accuracy below 90%.

You can only adjust model architecture and training hyperparameters.

You cannot change the dataset or add more data.

You must keep the entity types limited to PERSON, ORG, LOC, and DATE.

Hint 1

Hint 2

Hint 3

Hint 4

Solution

NLP

import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Embedding, Bidirectional, LSTM, Dense, Dropout
from tensorflow.keras.callbacks import EarlyStopping

# Assume X_train, y_train, X_val, y_val are preprocessed and ready

vocab_size = 10000  # example vocabulary size
embedding_dim = 64
max_len = 100  # max length of input sequences
num_classes = 5  # 4 entity types + 1 for 'O' (no entity)

model = Sequential([
    Embedding(vocab_size, embedding_dim, input_length=max_len),
    Bidirectional(LSTM(64, return_sequences=True)),
    Dropout(0.5),
    Dense(64, activation='relu'),
    Dropout(0.5),
    Dense(num_classes, activation='softmax')
])

model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.0005),
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

early_stop = EarlyStopping(monitor='val_loss', patience=3, restore_best_weights=True)

history = model.fit(X_train, y_train,
                    epochs=20,
                    batch_size=32,
                    validation_data=(X_val, y_val),
                    callbacks=[early_stop])

Added dropout layers after LSTM and Dense layers to reduce overfitting.

Lowered learning rate to 0.0005 for smoother convergence.

Added early stopping to stop training when validation loss stops improving.

Results Interpretation

Before: Training accuracy: 92%, Validation accuracy: 75%, Validation F1-score: 0.70

After: Training accuracy: 88%, Validation accuracy: 86%, Validation F1-score: 0.82

Adding dropout and early stopping helped reduce overfitting, improving validation accuracy and F1-score while slightly lowering training accuracy. This shows how controlling model complexity and training duration helps models generalize better.

Bonus Experiment

Try using a pretrained language model like BERT for named entity recognition on the same dataset.

💡 Hint

Use a pretrained transformer model with a token classification head and fine-tune it on your entity dataset for potentially better accuracy.