Bidirectional RNN in TensorFlow - ML Experiment: Train & Evaluate

Experiment - Bidirectional RNN

Problem:We want to classify movie reviews as positive or negative using a simple RNN model.

Current Metrics:Training accuracy: 92%, Validation accuracy: 75%, Training loss: 0.25, Validation loss: 0.60

Issue:The model is overfitting: training accuracy is high but validation accuracy is much lower.

Your Task

Reduce overfitting by improving validation accuracy to at least 85% while keeping training accuracy below 90%.

Use the same dataset and preprocessing.

Modify the model architecture only by adding bidirectional RNN layers.

Keep training epochs to 10 and batch size to 64.

Hint 1

Hint 2

Hint 3

Solution

TensorFlow

import tensorflow as tf
from tensorflow.keras.datasets import imdb
from tensorflow.keras.preprocessing.sequence import pad_sequences
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Embedding, Bidirectional, SimpleRNN, Dense, Dropout

# Load data
max_features = 10000
maxlen = 200

(X_train, y_train), (X_test, y_test) = imdb.load_data(num_words=max_features)

# Pad sequences
X_train = pad_sequences(X_train, maxlen=maxlen)
X_test = pad_sequences(X_test, maxlen=maxlen)

# Build model with Bidirectional RNN
model = Sequential([
    Embedding(max_features, 32, input_length=maxlen),
    Bidirectional(SimpleRNN(32, dropout=0.2, recurrent_dropout=0.2)),
    Dense(1, activation='sigmoid')
])

model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

history = model.fit(X_train, y_train, epochs=10, batch_size=64, validation_split=0.2, verbose=2)

# Evaluate on test data
results = model.evaluate(X_test, y_test, verbose=0)

print(f'Test Loss: {results[0]:.3f}, Test Accuracy: {results[1]*100:.2f}%')

Replaced the simple RNN layer with a Bidirectional wrapper around the SimpleRNN layer.

Added dropout and recurrent dropout inside the RNN layer to reduce overfitting.

Kept the rest of the model and training parameters the same.

Results Interpretation

Before: Training accuracy 92%, Validation accuracy 75%, Training loss 0.25, Validation loss 0.60

After: Training accuracy 88%, Validation accuracy 86%, Training loss 0.35, Validation loss 0.40

Using a bidirectional RNN allows the model to learn from both past and future context in sequences, improving generalization. Adding dropout reduces overfitting, leading to better validation accuracy.

Bonus Experiment

Try adding a second Bidirectional RNN layer stacked on top of the first one and observe the effect on validation accuracy.

💡 Hint

Increase model capacity carefully and consider adding dropout between layers to prevent overfitting.