RNN-based text generation in NLP - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding RNN hidden state behavior

In an RNN used for text generation, what does the hidden state represent during training?

A It contains the loss value used to update the model weights.

B It holds the final output probabilities for the next character prediction.

C It resets to zero after every character to avoid memory buildup.

D It stores information about previous characters or words to influence future predictions.

💡 Hint

Think about how RNNs remember context from earlier inputs.

The hidden state in an RNN acts like a memory that carries information from previous inputs, helping the model understand context when predicting the next character or word.

❓ Predict Output

intermediate

Output shape of RNN layer in text generation

What is the shape of the output tensor from an RNN layer when processing a batch of sequences with shape (batch_size=4, sequence_length=10, input_dim=8) and hidden size 16?

NLP

import torch
import torch.nn as nn
rnn = nn.RNN(input_size=8, hidden_size=16, batch_first=True)
inputs = torch.randn(4, 10, 8)
output, hidden = rnn(inputs)
print(output.shape)

A torch.Size([10, 4, 16])

B torch.Size([4, 10, 16])

C torch.Size([4, 16])

D torch.Size([4, 10, 8])

💡 Hint

Remember batch_first=True means batch is the first dimension.

The RNN output has shape (batch_size, sequence_length, hidden_size) when batch_first=True. Here, batch_size=4, sequence_length=10, hidden_size=16.

❓ Hyperparameter

advanced

Choosing sequence length for training RNN text generation

Which sequence length is generally better for training an RNN text generator to capture long-term dependencies without causing too much memory use?

A Sequence length does not affect RNN training or memory.

B Very long sequences (e.g., 1000 characters) to capture all context at once.

C Moderate length sequences (e.g., 50-100 characters) balancing context and memory.

D Very short sequences (e.g., 5 characters) to speed up training.

💡 Hint

Think about the trade-off between context and computational resources.

Moderate sequence lengths allow the RNN to learn useful context without excessive memory use or vanishing gradients that happen with very long sequences.

❓ Metrics

advanced

Evaluating RNN text generation with perplexity

What does a lower perplexity score indicate when evaluating an RNN text generation model?

A The model is more confident and accurate in predicting the next character or word.

B The model is overfitting and memorizing the training data.

C The model has a higher loss value during training.

D The model is underfitting and not learning the data patterns.

💡 Hint

Perplexity measures how well the model predicts the next token.

Lower perplexity means the model assigns higher probabilities to the correct next tokens, showing better prediction performance.

🔧 Debug

expert

Identifying cause of exploding gradients in RNN training

During training an RNN for text generation, the loss suddenly becomes NaN and the model weights explode. Which code snippet is the most likely cause?

NLP

optimizer = torch.optim.Adam(model.parameters(), lr=1.0)
for inputs, targets in dataloader:
    optimizer.zero_grad()
    outputs = model(inputs)
    loss = loss_fn(outputs, targets)
    loss.backward()
    optimizer.step()

A The learning rate is too high (lr=1.0), causing unstable updates.

B The optimizer.zero_grad() is missing before loss.backward().

C The loss function is not differentiable, causing NaN gradients.

D The model inputs are not normalized, causing exploding gradients.

💡 Hint

Check the learning rate value and its effect on training stability.

A very high learning rate like 1.0 can cause large weight updates, leading to exploding gradients and NaN loss values.

Practice

1. What is the main purpose of using an RNN in text generation?

easy

A. To count the number of words in a sentence

B. To sort words alphabetically

C. To translate text into another language

D. To learn patterns in sequences of words to predict the next word

Solution

1. Step 1: Understand RNN function in text

   RNNs process sequences step-by-step, remembering past words to predict what comes next.

2. Step 2: Identify the goal of text generation

   The goal is to generate new text by predicting the next word based on learned patterns.

3. Final Answer:

   To learn patterns in sequences of words to predict the next word -> Option D

4. Quick Check:

   RNN predicts next word in sequence = C [OK]

Hint: RNNs remember word order to guess the next word [OK]

Common Mistakes:

• Thinking RNNs just count words
• Confusing RNNs with sorting algorithms
• Assuming RNNs translate text directly

2. Which of the following is the correct way to define an embedding layer in a Keras RNN model for text generation?

easy

A. Embedding(input_length=64, input_dim=10, output_dim=1000)

B. Embedding(output_dim=1000, input_dim=64, input_length=10)

C. Embedding(input_dim=1000, output_dim=64, input_length=10)

D. Embedding(input_dim=10, output_dim=1000, input_length=64)

Solution

1. Step 1: Recall embedding layer parameters

   Embedding layers require input_dim (vocab size), output_dim (embedding size), and input_length (sequence length).

2. Step 2: Match parameters correctly

   Embedding(input_dim=1000, output_dim=64, input_length=10) correctly sets input_dim=1000 (vocab size), output_dim=64 (embedding size), input_length=10 (sequence length).

3. Final Answer:

   Embedding(input_dim=1000, output_dim=64, input_length=10) -> Option C

4. Quick Check:

   Embedding(input_dim, output_dim, input_length) = A [OK]

Hint: Input_dim = vocab size, output_dim = embedding size [OK]

Common Mistakes:

• Swapping input_dim and output_dim
• Confusing input_length with output_dim
• Using wrong parameter names

3. Given this code snippet for training an RNN text generator, what will be the shape of the input data X if the vocabulary size is 5000, sequence length is 20, and batch size is 32?

model = Sequential()
model.add(Embedding(input_dim=5000, output_dim=50, input_length=20))
model.add(SimpleRNN(100))
model.add(Dense(5000, activation='softmax'))

X = np.random.randint(0, 5000, (32, 20))

medium

A. (20, 32)

B. (32, 20)

C. (32, 50)

D. (5000, 20)

Solution

1. Step 1: Understand input shape for embedding

   The input to the embedding layer is a 2D array: (batch_size, sequence_length).

2. Step 2: Check given data shape

   X is created with shape (32, 20), matching batch size 32 and sequence length 20.

3. Final Answer:

   (32, 20) -> Option B

4. Quick Check:

   Input shape = (batch_size, sequence_length) = (32, 20) [OK]

Hint: Input shape = batch size by sequence length [OK]

Common Mistakes:

• Confusing embedding output shape with input shape
• Swapping batch size and sequence length
• Assuming embedding changes input shape

4. You wrote this code to train an RNN text generator but get a shape mismatch error:

model = Sequential()
model.add(Embedding(input_dim=10000, output_dim=64, input_length=15))
model.add(SimpleRNN(128))
model.add(Dense(10000, activation='softmax'))

X = np.random.randint(0, 10000, (64, 15))
y = np.random.randint(0, 10000, (64, 15))  # target labels

model.compile(loss='sparse_categorical_crossentropy', optimizer='adam')
model.fit(X, y, epochs=5)

What is the main issue causing the error?

medium

A. Target labels y should be shape (64,) with integer word indices, not (64, 15)

B. Embedding input_dim is too large

C. SimpleRNN units should match output_dim of embedding

D. Loss function sparse_categorical_crossentropy is incorrect

Solution

1. Step 1: Check target label shape for next word prediction

   For next word prediction, y should be a 1D array of word indices (batch_size,), not sequences.

2. Step 2: Identify mismatch in y shape

   y has shape (64, 15), which causes shape mismatch with model output (64, 10000).

3. Final Answer:

   Target labels y should be shape (64,) with integer word indices, not (64, 15) -> Option A

4. Quick Check:

   y shape must match output shape = B [OK]

Hint: Targets for next word are 1D, not sequences [OK]

Common Mistakes:

• Using sequences as targets instead of next word
• Confusing embedding size with RNN units
• Changing loss function unnecessarily

5. You want to generate text using a trained RNN model. Which approach correctly generates text word by word after training?

hard

A. Feed the model the initial seed sequence, predict the next word, append it, then use the updated sequence to predict again

B. Feed the entire training dataset at once to get all generated words

C. Use the model to predict all words simultaneously without updating input

D. Randomly select words from the vocabulary without using the model

Solution

1. Step 1: Understand sequential generation

   Text generation uses the model to predict one word at a time, updating input with new words.

2. Step 2: Identify correct iterative approach

   Feed the model the initial seed sequence, predict the next word, append it, then use the updated sequence to predict again describes feeding seed, predicting next word, appending it, and repeating, which is correct.

3. Final Answer:

   Feed the model the initial seed sequence, predict the next word, append it, then use the updated sequence to predict again -> Option A

4. Quick Check:

   Generate word-by-word with updated input = D [OK]

Hint: Generate text stepwise, updating input each time [OK]

Common Mistakes:

• Trying to generate all words at once
• Ignoring the need to update input sequence
• Selecting words randomly without model