Sentiment with context (sarcasm, negation) in NLP - ML Experiment: Train & Evaluate

Experiment - Sentiment with context (sarcasm, negation)

Problem:We want to classify the sentiment of text messages, including tricky cases like sarcasm and negation. The current model predicts sentiment but often gets sarcastic or negated sentences wrong.

Current Metrics:Training accuracy: 92%, Validation accuracy: 70%, Validation loss: 0.85

Issue:The model overfits the training data and performs poorly on validation, especially on sarcastic and negated sentences.

Your Task

Reduce overfitting and improve validation accuracy to at least 80%, especially on sarcastic and negated sentences, while keeping training accuracy below 90%.

You cannot change the dataset or add more data.

You must keep the same model architecture type (a simple LSTM-based classifier).

You can only adjust hyperparameters and add regularization techniques.

Hint 1

Hint 2

Hint 3

Hint 4

Solution

NLP

import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Embedding, LSTM, Dense, Dropout
from tensorflow.keras.callbacks import EarlyStopping

# Assume X_train, y_train, X_val, y_val are preprocessed and ready
vocab_size = 10000
embedding_dim = 64
max_length = 100

model = Sequential([
    Embedding(vocab_size, embedding_dim, input_length=max_length),
    LSTM(64, return_sequences=False),
    Dropout(0.5),
    Dense(32, activation='relu', kernel_regularizer=tf.keras.regularizers.l2(0.01)),
    Dropout(0.5),
    Dense(1, activation='sigmoid')
])

model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.0005),
              loss='binary_crossentropy',
              metrics=['accuracy'])

early_stop = EarlyStopping(monitor='val_loss', patience=3, restore_best_weights=True)

history = model.fit(X_train, y_train,
                    epochs=20,
                    batch_size=32,
                    validation_data=(X_val, y_val),
                    callbacks=[early_stop])

Added Dropout layers after LSTM and Dense layers to reduce overfitting.

Added L2 regularization to Dense layer to penalize large weights.

Lowered learning rate from default to 0.0005 for smoother training.

Added EarlyStopping callback to stop training when validation loss stops improving.

Results Interpretation

Before: Training accuracy 92%, Validation accuracy 70%, Validation loss 0.85

After: Training accuracy 88%, Validation accuracy 82%, Validation loss 0.65

Adding dropout and L2 regularization helped reduce overfitting, improving validation accuracy especially on difficult cases like sarcasm and negation. Early stopping prevented overtraining, and lowering learning rate helped the model converge better.

Bonus Experiment

Try using a pretrained transformer model like BERT fine-tuned on this sentiment task to better capture sarcasm and negation.

💡 Hint

Use Hugging Face's transformers library and fine-tune a small BERT model with a classification head.

Practice

1. What effect does the word not usually have on sentiment in a sentence?

easy

A. It makes the sentence neutral

B. It always makes the sentence positive

C. It has no effect on sentiment

D. It reverses the sentiment of the following phrase

Solution

1. Step 1: Understand negation in sentiment

   The word not is a negation word that flips the meaning of the phrase it modifies.

2. Step 2: Apply to sentiment analysis

   If a phrase is positive, adding not before it usually makes it negative, and vice versa.

3. Final Answer:

   It reverses the sentiment of the following phrase -> Option D

4. Quick Check:

   Negation flips sentiment = A [OK]

Hint: Negation words flip sentiment meaning quickly [OK]

Common Mistakes:

• Ignoring negation words in sentiment
• Assuming negation always makes positive
• Treating negation as neutral

2. Which of the following is the correct way to handle negation in a simple sentiment analysis code snippet?

easy

A. Ignore negation words and analyze sentiment word by word

B. Flip sentiment polarity of words following negation words

C. Treat negation words as positive sentiment

D. Remove negation words before analysis

Solution

1. Step 1: Identify negation handling in code

   Proper handling means detecting negation words and flipping sentiment of words after them.

2. Step 2: Evaluate options

   Ignoring or removing negation loses meaning; treating negation as positive is wrong.

3. Final Answer:

   Flip sentiment polarity of words following negation words -> Option B

4. Quick Check:

   Flip sentiment after negation = B [OK]

Hint: Flip sentiment after negation words in code [OK]

Common Mistakes:

• Ignoring negation in code
• Removing negation words blindly
• Misclassifying negation as positive

3. Consider this Python code snippet for sentiment scoring with negation handling:

sentence = "I do not like this movie"
words = sentence.split()
sentiment_dict = {"like": 1, "movie": 0}
score = 0
negate = False
for w in words:
    if w == "not":
        negate = True
        continue
    val = sentiment_dict.get(w, 0)
    if negate:
        val = -val
        negate = False
    score += val
print(score)

What is the printed output?

medium

A. -1

B. 0

C. 1

D. 2

Solution

1. Step 1: Trace the loop and negation flag

   Words: ['I', 'do', 'not', 'like', 'this', 'movie'] - 'not' sets negate=True - Next word 'like' has sentiment 1, negated to -1 - 'movie' sentiment 0, no negation - Other words have 0 sentiment

2. Step 2: Calculate total score

   Score = -1 (from 'like') + 0 (from 'movie') + 0 (others) = -1

3. Final Answer:

   -1 -> Option A

4. Quick Check:

   Negation flips 1 to -1 = A [OK]

Hint: Negation flips next word sentiment once [OK]

Common Mistakes:

• Not resetting negate flag after one word
• Ignoring words not in sentiment_dict
• Assuming negation affects all following words

4. The following code tries to handle negation but gives wrong sentiment scores:

sentence = "I am not happy"
words = sentence.split()
sentiment_dict = {"happy": 1}
score = 0
negate = False
for w in words:
    if w == "not":
        negate = True
    val = sentiment_dict.get(w, 0)
    if negate:
        val = -val
    score += val
print(score)

What is the main bug causing incorrect output?

medium

A. Negation flag is never reset after use

B. Sentiment dictionary missing 'not' key

C. Loop skips words after 'not'

D. Score is not initialized to zero

Solution

1. Step 1: Analyze negation flag usage

   Negate is set True on 'not' but never reset to False, so all following words are negated.

2. Step 2: Understand impact on sentiment score

   All words after 'not' get negated, causing wrong total sentiment.

3. Final Answer:

   Negation flag is never reset after use -> Option A

4. Quick Check:

   Negate flag reset missing = C [OK]

Hint: Reset negation flag after negating one word [OK]

Common Mistakes:

• Forgetting to reset negation flag
• Adding keys unnecessarily to sentiment dict
• Assuming loop skips words after negation

5. You want to improve a sentiment model to detect sarcasm, which often reverses sentiment meaning. Which approach is best to handle sarcasm in sentiment analysis?

hard

A. Use a simple bag-of-words model ignoring word order

B. Assign fixed positive sentiment to all sarcastic sentences

C. Add a sarcasm detection module using context and tone features

D. Remove all negation words from the text before analysis

Solution

1. Step 1: Understand sarcasm complexity

   Sarcasm changes sentiment meaning and needs context, tone, or special features beyond simple word counts.

2. Step 2: Evaluate approaches

   Simple bag-of-words or removing negations lose sarcasm cues; fixed positive sentiment is incorrect.

3. Final Answer:

   Add a sarcasm detection module using context and tone features -> Option C

4. Quick Check:

   Sarcasm needs special detection = D [OK]

Hint: Detect sarcasm with context and tone features [OK]

Common Mistakes:

• Using simple models ignoring sarcasm
• Removing negation words blindly
• Assigning fixed sentiment to sarcasm