NLP applications in real world - ML Experiment: Train & Evaluate

Experiment - NLP applications in real world

Problem:You have a text classification model that categorizes customer reviews into positive or negative sentiment. The model currently performs well on training data but poorly on new reviews, showing signs of overfitting.

Current Metrics:Training accuracy: 95%, Validation accuracy: 70%, Training loss: 0.15, Validation loss: 0.65

Issue:The model overfits the training data, causing low validation accuracy and poor generalization to new reviews.

Your Task

Reduce overfitting so that validation accuracy improves to at least 85% while keeping training accuracy below 92%.

You can only modify the model architecture and training parameters.

You cannot change the dataset or add more data.

Hint 1

Hint 2

Hint 3

Solution

NLP

import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Embedding, LSTM, Dense, Dropout
from tensorflow.keras.callbacks import EarlyStopping

# Sample data placeholders (replace with actual data loading)
X_train, y_train = ...  # training data
X_val, y_val = ...      # validation data

model = Sequential([
    Embedding(input_dim=10000, output_dim=64, input_length=100),
    LSTM(64, return_sequences=False),
    Dropout(0.5),
    Dense(1, activation='sigmoid')
])

model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),
              loss='binary_crossentropy',
              metrics=['accuracy'])

early_stop = EarlyStopping(monitor='val_loss', patience=3, restore_best_weights=True)

history = model.fit(X_train, y_train,
                    epochs=20,
                    batch_size=32,
                    validation_data=(X_val, y_val),
                    callbacks=[early_stop])

Added a Dropout layer with rate 0.5 after the LSTM layer to reduce overfitting.

Implemented EarlyStopping callback to stop training when validation loss stops improving.

Set learning rate to 0.001 for stable training.

Results Interpretation

Before: Training accuracy 95%, Validation accuracy 70%, Training loss 0.15, Validation loss 0.65

After: Training accuracy 90%, Validation accuracy 87%, Training loss 0.30, Validation loss 0.40

Adding dropout and early stopping helps reduce overfitting, improving validation accuracy and model generalization.

Bonus Experiment

Try using a pretrained language model like BERT for the same classification task to see if it improves accuracy further.

💡 Hint

Use transfer learning with a pretrained BERT model and fine-tune it on your dataset.