RNN for text classification in NLP

Introduction

RNNs help computers understand sentences by reading words one by one. This helps to decide what category a text belongs to, like spam or not spam.

When you want to tell if an email is spam or not by reading its words.

When you want to find out if a movie review is positive or negative.

When you want to sort news articles into topics like sports or politics.

When you want to detect the mood of a tweet, like happy or sad.

Syntax

NLP

model = Sequential()
model.add(Embedding(input_dim=vocab_size, output_dim=embedding_dim, input_length=max_length))
model.add(SimpleRNN(units=hidden_units))
model.add(Dense(units=num_classes, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

Embedding turns words into numbers the model can understand.

SimpleRNN reads the words one by one to learn the order and meaning.

Examples

This example builds a small RNN for classifying texts into 2 categories.

NLP

model = Sequential()
model.add(Embedding(1000, 64, input_length=10))
model.add(SimpleRNN(32))
model.add(Dense(2, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

This example uses a bigger vocabulary and longer texts for 3 classes.

NLP

model = Sequential()
model.add(Embedding(5000, 128, input_length=20))
model.add(SimpleRNN(64))
model.add(Dense(3, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

Sample Model

This program trains a simple RNN to classify 6 short texts into 2 groups. It shows training accuracy and predicted classes.

NLP

import numpy as np
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Embedding, SimpleRNN, Dense
from tensorflow.keras.utils import to_categorical

# Sample data: 6 texts, each with 5 words (word indexes)
x_train = np.array([
    [1, 2, 3, 4, 0],
    [2, 3, 4, 5, 0],
    [1, 3, 5, 0, 0],
    [4, 5, 6, 7, 0],
    [5, 6, 7, 8, 0],
    [6, 7, 8, 9, 0]
])

# Labels: 2 classes (0 or 1)
y_train = np.array([0, 0, 0, 1, 1, 1])
y_train_cat = to_categorical(y_train, num_classes=2)

vocab_size = 10  # words numbered 0-9
embedding_dim = 8
max_length = 5
hidden_units = 4
num_classes = 2

model = Sequential()
model.add(Embedding(input_dim=vocab_size, output_dim=embedding_dim, input_length=max_length))
model.add(SimpleRNN(units=hidden_units))
model.add(Dense(units=num_classes, activation='softmax'))

model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

# Train model
history = model.fit(x_train, y_train_cat, epochs=10, verbose=0)

# Predict on training data
predictions = model.predict(x_train)
predicted_classes = np.argmax(predictions, axis=1)

print(f"Training accuracy: {history.history['accuracy'][-1]:.2f}")
print(f"Predicted classes: {predicted_classes.tolist()}")

Important Notes

RNNs read words in order, so they understand sentence flow better than simple models.

Embedding layer helps convert words into numbers that keep their meaning.

Training on small data is just for learning; real tasks need more data for good results.

Summary

RNNs are good for reading text word by word to classify it.

Embedding layers turn words into numbers the model can learn from.

SimpleRNN layer helps the model remember word order and context.

Practice

1. What is the main reason to use an RNN (Recurrent Neural Network) for text classification tasks?

easy

A. Because RNNs only work with images

B. Because RNNs are faster than other neural networks

C. Because RNNs do not require any training data

D. Because RNNs can remember the order of words and context in sentences

Solution

1. Step 1: Understand RNN's role in text

   RNNs process sequences of words one by one, keeping track of previous words to understand context.

2. Step 2: Identify why order matters

   Text meaning depends on word order, and RNNs remember this order, unlike simple models.

3. Final Answer:

   Because RNNs can remember the order of words and context in sentences -> Option D

4. Quick Check:

   RNN remembers sequence = D [OK]

Hint: RNNs are for sequences and context, not speed or images [OK]

Common Mistakes:

• Thinking RNNs are faster than other models
• Believing RNNs don't need training data
• Confusing RNNs with image-only models

2. Which of the following is the correct way to add a SimpleRNN layer with 32 units in Keras for text classification?

easy

A. model.add(SimpleRNN(32, input_shape=(None, 100)))

B. model.add(SimpleRNN(units=32))

C. model.add(SimpleRNN(32))

D. model.add(SimpleRNN(32, activation='relu'))

Solution

1. Step 1: Recall SimpleRNN syntax

   SimpleRNN requires number of units and input shape for the first layer in a model.

2. Step 2: Check options for correct usage

   model.add(SimpleRNN(32, input_shape=(None, 100))) correctly specifies 32 units and input shape (sequence length unknown, 100 features).

3. Final Answer:

   model.add(SimpleRNN(32, input_shape=(None, 100))) -> Option A

4. Quick Check:

   SimpleRNN needs units and input shape first layer = A [OK]

Hint: First RNN layer needs input_shape, else error [OK]

Common Mistakes:

• Omitting input_shape in first RNN layer
• Using activation='relu' instead of default tanh
• Passing units as keyword incorrectly

3. Given this Keras model snippet for text classification:

model = Sequential()
model.add(Embedding(input_dim=5000, output_dim=16, input_length=10))
model.add(SimpleRNN(8))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

history = model.fit(X_train, y_train, epochs=2, batch_size=32)
print(history.history['accuracy'][-1])

What does history.history['accuracy'][-1] represent?

medium

A. The accuracy of the model on the entire training data after the last epoch

B. The accuracy of the model on the last training batch of the last epoch

C. The loss value of the model after the last epoch

D. The accuracy of the model on the validation data after the last epoch

Solution

1. Step 1: Understand Keras history object

   history.history['accuracy'] stores training accuracy per epoch, so last element is final epoch training accuracy.

2. Step 2: Differentiate training vs batch vs validation

   It is training accuracy on all training data after last epoch, not batch or validation accuracy.

3. Final Answer:

   The accuracy of the model on the entire training data after the last epoch -> Option A

4. Quick Check:

   history.history['accuracy'][-1] = final training accuracy [OK]

Hint: history.history['accuracy'] is training accuracy per epoch [OK]

Common Mistakes:

• Confusing batch accuracy with epoch accuracy
• Mixing loss and accuracy values
• Assuming validation accuracy without validation data

4. You wrote this code to build an RNN model for text classification but get an error:

model = Sequential()
model.add(SimpleRNN(16))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

What is the most likely cause of the error?

medium

A. Dense layer cannot have sigmoid activation

B. SimpleRNN units must be 32 or more

C. Missing input shape for the first SimpleRNN layer

D. Loss function 'binary_crossentropy' is invalid

Solution

1. Step 1: Check first layer requirements

   The first RNN layer must know input shape to accept data; missing input_shape causes error.

2. Step 2: Validate other options

   Sigmoid activation in Dense is valid for binary classification; units can be any positive integer; binary_crossentropy is valid loss.

3. Final Answer:

   Missing input shape for the first SimpleRNN layer -> Option C

4. Quick Check:

   First RNN layer needs input_shape = B [OK]

Hint: Always set input_shape in first RNN layer to avoid errors [OK]

Common Mistakes:

• Assuming activation or loss function causes error
• Thinking units must be 32 or more
• Ignoring input shape requirement

5. You want to improve your RNN text classifier by adding an Embedding layer before the SimpleRNN. Which of these changes is correct and why?

Original:
model = Sequential()
model.add(SimpleRNN(16, input_shape=(10, 100)))
model.add(Dense(1, activation='sigmoid'))

Change:
model = Sequential()
model.add(Embedding(input_dim=5000, output_dim=100, input_length=10))
model.add(SimpleRNN(16))
model.add(Dense(1, activation='sigmoid'))

hard

A. Incorrect: Embedding output_dim must match SimpleRNN units

B. Correct: Embedding converts word indices to vectors, so SimpleRNN input shape changes automatically

C. Incorrect: Embedding layer should come after SimpleRNN

D. Incorrect: Embedding layer requires activation='relu'

Solution

1. Step 1: Understand Embedding role

   Embedding layer converts integer word indices into dense vectors, preparing input for RNN.

2. Step 2: Check model order and shapes

   Embedding outputs shape (batch, sequence_length, output_dim), matching SimpleRNN expected input shape, so no input_shape needed in SimpleRNN.

3. Final Answer:

   Correct: Embedding converts word indices to vectors, so SimpleRNN input shape changes automatically -> Option B

4. Quick Check:

   Embedding before RNN changes input shape correctly = C [OK]

Hint: Embedding layer must come before RNN to convert words to vectors [OK]

Common Mistakes:

• Placing Embedding after RNN
• Matching output_dim to RNN units incorrectly
• Adding activation to Embedding layer