RNN for text classification in NLP - Cheat Sheet & Quick Revision

Recall & Review

beginner

What does RNN stand for and what is its main use in text classification?

RNN stands for Recurrent Neural Network. It is mainly used to process sequences like text by remembering previous words to understand context for classification.

beginner

How does an RNN handle input sequences differently from a regular neural network?

An RNN processes input one step at a time and keeps a memory of previous steps, allowing it to understand order and context in sequences like sentences.

intermediate

What is the role of the hidden state in an RNN?

The hidden state stores information from previous inputs in the sequence, helping the RNN remember context as it processes each new word.

intermediate

Why might a simple RNN struggle with long text sequences?

Simple RNNs can forget information from earlier in the sequence due to problems like vanishing gradients, making it hard to learn long-range dependencies.

intermediate

Name two advanced RNN variants that help with remembering long-term context.

Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) are two RNN variants designed to better remember long-term information in sequences.

What is the main advantage of using an RNN for text classification?

A It uses convolution to detect features.

B It processes all words independently without order.

C It can remember previous words to understand context.

D It only works with fixed-length inputs.

RNNs process sequences step-by-step and keep memory of previous inputs, which helps understand context in text.

What problem do LSTM and GRU solve in RNNs?

A They speed up training by skipping steps.

B They help remember information over long sequences.

C They reduce the size of the input data.

D They convert text into images.

LSTM and GRU are designed to keep important information longer and avoid forgetting it during training.

Which part of an RNN holds information from previous words?

A Hidden state

B Input embedding

C Output layer

D Loss function

The hidden state stores information from earlier inputs to help the RNN understand context.

Why can simple RNNs struggle with very long text sequences?

A They cannot process sequences longer than 10 words.

B They only work with numeric data.

C They require labeled images for training.

D They forget earlier information due to vanishing gradients.

Vanishing gradients cause simple RNNs to lose information from earlier in long sequences.

In text classification with RNNs, what is the typical final step?

A Use the last hidden state to predict the class.

B Convert text to images.

C Ignore the sequence order.

D Use convolution layers to classify.

The last hidden state summarizes the sequence and is used to predict the text's class.

Explain how an RNN processes a sentence for text classification.

Think about how the network reads words one by one and remembers what it saw before.

Describe why LSTM or GRU might be better than a simple RNN for long text sequences.

Consider how these variants help keep important information longer.

Practice

1. What is the main reason to use an RNN (Recurrent Neural Network) for text classification tasks?

easy

A. Because RNNs only work with images

B. Because RNNs are faster than other neural networks

C. Because RNNs do not require any training data

D. Because RNNs can remember the order of words and context in sentences

Solution

1. Step 1: Understand RNN's role in text

   RNNs process sequences of words one by one, keeping track of previous words to understand context.

2. Step 2: Identify why order matters

   Text meaning depends on word order, and RNNs remember this order, unlike simple models.

3. Final Answer:

   Because RNNs can remember the order of words and context in sentences -> Option D

4. Quick Check:

   RNN remembers sequence = D [OK]

Hint: RNNs are for sequences and context, not speed or images [OK]

Common Mistakes:

• Thinking RNNs are faster than other models
• Believing RNNs don't need training data
• Confusing RNNs with image-only models

2. Which of the following is the correct way to add a SimpleRNN layer with 32 units in Keras for text classification?

easy

A. model.add(SimpleRNN(32, input_shape=(None, 100)))

B. model.add(SimpleRNN(units=32))

C. model.add(SimpleRNN(32))

D. model.add(SimpleRNN(32, activation='relu'))

Solution

1. Step 1: Recall SimpleRNN syntax

   SimpleRNN requires number of units and input shape for the first layer in a model.

2. Step 2: Check options for correct usage

   model.add(SimpleRNN(32, input_shape=(None, 100))) correctly specifies 32 units and input shape (sequence length unknown, 100 features).

3. Final Answer:

   model.add(SimpleRNN(32, input_shape=(None, 100))) -> Option A

4. Quick Check:

   SimpleRNN needs units and input shape first layer = A [OK]

Hint: First RNN layer needs input_shape, else error [OK]

Common Mistakes:

• Omitting input_shape in first RNN layer
• Using activation='relu' instead of default tanh
• Passing units as keyword incorrectly

3. Given this Keras model snippet for text classification:

model = Sequential()
model.add(Embedding(input_dim=5000, output_dim=16, input_length=10))
model.add(SimpleRNN(8))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

history = model.fit(X_train, y_train, epochs=2, batch_size=32)
print(history.history['accuracy'][-1])

What does history.history['accuracy'][-1] represent?

medium

A. The accuracy of the model on the entire training data after the last epoch

B. The accuracy of the model on the last training batch of the last epoch

C. The loss value of the model after the last epoch

D. The accuracy of the model on the validation data after the last epoch

Solution

1. Step 1: Understand Keras history object

   history.history['accuracy'] stores training accuracy per epoch, so last element is final epoch training accuracy.

2. Step 2: Differentiate training vs batch vs validation

   It is training accuracy on all training data after last epoch, not batch or validation accuracy.

3. Final Answer:

   The accuracy of the model on the entire training data after the last epoch -> Option A

4. Quick Check:

   history.history['accuracy'][-1] = final training accuracy [OK]

Hint: history.history['accuracy'] is training accuracy per epoch [OK]

Common Mistakes:

• Confusing batch accuracy with epoch accuracy
• Mixing loss and accuracy values
• Assuming validation accuracy without validation data

4. You wrote this code to build an RNN model for text classification but get an error:

model = Sequential()
model.add(SimpleRNN(16))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

What is the most likely cause of the error?

medium

A. Dense layer cannot have sigmoid activation

B. SimpleRNN units must be 32 or more

C. Missing input shape for the first SimpleRNN layer

D. Loss function 'binary_crossentropy' is invalid

Solution

1. Step 1: Check first layer requirements

   The first RNN layer must know input shape to accept data; missing input_shape causes error.

2. Step 2: Validate other options

   Sigmoid activation in Dense is valid for binary classification; units can be any positive integer; binary_crossentropy is valid loss.

3. Final Answer:

   Missing input shape for the first SimpleRNN layer -> Option C

4. Quick Check:

   First RNN layer needs input_shape = B [OK]

Hint: Always set input_shape in first RNN layer to avoid errors [OK]

Common Mistakes:

• Assuming activation or loss function causes error
• Thinking units must be 32 or more
• Ignoring input shape requirement

5. You want to improve your RNN text classifier by adding an Embedding layer before the SimpleRNN. Which of these changes is correct and why?

Original:
model = Sequential()
model.add(SimpleRNN(16, input_shape=(10, 100)))
model.add(Dense(1, activation='sigmoid'))

Change:
model = Sequential()
model.add(Embedding(input_dim=5000, output_dim=100, input_length=10))
model.add(SimpleRNN(16))
model.add(Dense(1, activation='sigmoid'))

hard

A. Incorrect: Embedding output_dim must match SimpleRNN units

B. Correct: Embedding converts word indices to vectors, so SimpleRNN input shape changes automatically

C. Incorrect: Embedding layer should come after SimpleRNN

D. Incorrect: Embedding layer requires activation='relu'

Solution

1. Step 1: Understand Embedding role

   Embedding layer converts integer word indices into dense vectors, preparing input for RNN.

2. Step 2: Check model order and shapes

   Embedding outputs shape (batch, sequence_length, output_dim), matching SimpleRNN expected input shape, so no input_shape needed in SimpleRNN.

3. Final Answer:

   Correct: Embedding converts word indices to vectors, so SimpleRNN input shape changes automatically -> Option B

4. Quick Check:

   Embedding before RNN changes input shape correctly = C [OK]

Hint: Embedding layer must come before RNN to convert words to vectors [OK]

Common Mistakes:

• Placing Embedding after RNN
• Matching output_dim to RNN units incorrectly
• Adding activation to Embedding layer