Sentiment with context (sarcasm, negation) in NLP - ML Experiment: Train & Evaluate

Experiment - Sentiment with context (sarcasm, negation)

Problem:We want to classify the sentiment of text messages, including tricky cases like sarcasm and negation. The current model predicts sentiment but often gets sarcastic or negated sentences wrong.

Current Metrics:Training accuracy: 92%, Validation accuracy: 70%, Validation loss: 0.85

Issue:The model overfits the training data and performs poorly on validation, especially on sarcastic and negated sentences.

Your Task

Reduce overfitting and improve validation accuracy to at least 80%, especially on sarcastic and negated sentences, while keeping training accuracy below 90%.

You cannot change the dataset or add more data.

You must keep the same model architecture type (a simple LSTM-based classifier).

You can only adjust hyperparameters and add regularization techniques.

Hint 1

Hint 2

Hint 3

Hint 4

Solution

NLP

import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Embedding, LSTM, Dense, Dropout
from tensorflow.keras.callbacks import EarlyStopping

# Assume X_train, y_train, X_val, y_val are preprocessed and ready
vocab_size = 10000
embedding_dim = 64
max_length = 100

model = Sequential([
    Embedding(vocab_size, embedding_dim, input_length=max_length),
    LSTM(64, return_sequences=False),
    Dropout(0.5),
    Dense(32, activation='relu', kernel_regularizer=tf.keras.regularizers.l2(0.01)),
    Dropout(0.5),
    Dense(1, activation='sigmoid')
])

model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.0005),
              loss='binary_crossentropy',
              metrics=['accuracy'])

early_stop = EarlyStopping(monitor='val_loss', patience=3, restore_best_weights=True)

history = model.fit(X_train, y_train,
                    epochs=20,
                    batch_size=32,
                    validation_data=(X_val, y_val),
                    callbacks=[early_stop])

Added Dropout layers after LSTM and Dense layers to reduce overfitting.

Added L2 regularization to Dense layer to penalize large weights.

Lowered learning rate from default to 0.0005 for smoother training.

Added EarlyStopping callback to stop training when validation loss stops improving.

Results Interpretation

Before: Training accuracy 92%, Validation accuracy 70%, Validation loss 0.85

After: Training accuracy 88%, Validation accuracy 82%, Validation loss 0.65

Adding dropout and L2 regularization helped reduce overfitting, improving validation accuracy especially on difficult cases like sarcasm and negation. Early stopping prevented overtraining, and lowering learning rate helped the model converge better.

Bonus Experiment

Try using a pretrained transformer model like BERT fine-tuned on this sentiment task to better capture sarcasm and negation.

💡 Hint

Use Hugging Face's transformers library and fine-tune a small BERT model with a classification head.