Experiment - RNN for text classification

Problem:Classify movie reviews as positive or negative using a simple RNN model.

Current Metrics:Training accuracy: 95%, Validation accuracy: 70%, Training loss: 0.15, Validation loss: 0.60

Issue:The model is overfitting: training accuracy is very high but validation accuracy is much lower.

Your Task

Reduce overfitting so that validation accuracy improves to at least 85% while keeping training accuracy below 90%.

Keep the RNN architecture simple (one recurrent layer).

Do not increase the dataset size.

Use only changes in model architecture or training parameters.

Hint 1

Hint 2

Hint 3

Hint 4

Hint 5

Solution

NLP

import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Embedding, SimpleRNN, Dense, Dropout
from tensorflow.keras.preprocessing.sequence import pad_sequences
from tensorflow.keras.datasets import imdb
from tensorflow.keras.callbacks import EarlyStopping

# Load data
max_features = 10000
maxlen = 100
(X_train, y_train), (X_test, y_test) = imdb.load_data(num_words=max_features)

# Pad sequences
X_train = pad_sequences(X_train, maxlen=maxlen)
X_test = pad_sequences(X_test, maxlen=maxlen)

# Build model with dropout and recurrent dropout
model = Sequential([
    Embedding(max_features, 32, input_length=maxlen),
    SimpleRNN(32, dropout=0.3, recurrent_dropout=0.3),
    Dropout(0.3),
    Dense(1, activation='sigmoid')
])

model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.0005),
              loss='binary_crossentropy',
              metrics=['accuracy'])

# Early stopping callback
early_stop = EarlyStopping(monitor='val_loss', patience=3, restore_best_weights=True)

# Train model
history = model.fit(X_train, y_train,
                    epochs=20,
                    batch_size=64,
                    validation_split=0.2,
                    callbacks=[early_stop],
                    verbose=2)

# Evaluate on test data
test_loss, test_acc = model.evaluate(X_test, y_test, verbose=0)

print(f'Test accuracy: {test_acc:.2f}', f'Test loss: {test_loss:.2f}')

Added dropout=0.3 and recurrent_dropout=0.3 to the SimpleRNN layer to reduce overfitting.

Added a Dropout layer after the RNN layer.

Reduced the number of units in the RNN from 64 (assumed original) to 32.

Added EarlyStopping callback to stop training when validation loss stops improving.

Lowered learning rate to 0.0005 for smoother training.

Results Interpretation

Before: Training accuracy 95%, Validation accuracy 70%, Training loss 0.15, Validation loss 0.60

After: Training accuracy 88%, Validation accuracy 86%, Training loss 0.30, Validation loss 0.40

Adding dropout and recurrent dropout reduces overfitting by preventing the model from relying too much on specific neurons. Early stopping helps avoid training too long. These changes improve validation accuracy and make the model generalize better.

Bonus Experiment

Try replacing the SimpleRNN layer with an LSTM layer and compare the validation accuracy.

💡 Hint

LSTM layers can capture longer dependencies in text and often improve performance on sequence tasks.