RNN-based text generation in NLP - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding RNN hidden state behavior

In an RNN used for text generation, what does the hidden state represent during training?

A It contains the loss value used to update the model weights.

B It holds the final output probabilities for the next character prediction.

C It resets to zero after every character to avoid memory buildup.

D It stores information about previous characters or words to influence future predictions.

💡 Hint

Think about how RNNs remember context from earlier inputs.

The hidden state in an RNN acts like a memory that carries information from previous inputs, helping the model understand context when predicting the next character or word.

❓ Predict Output

intermediate

Output shape of RNN layer in text generation

What is the shape of the output tensor from an RNN layer when processing a batch of sequences with shape (batch_size=4, sequence_length=10, input_dim=8) and hidden size 16?

NLP

import torch
import torch.nn as nn
rnn = nn.RNN(input_size=8, hidden_size=16, batch_first=True)
inputs = torch.randn(4, 10, 8)
output, hidden = rnn(inputs)
print(output.shape)

A torch.Size([10, 4, 16])

B torch.Size([4, 10, 16])

C torch.Size([4, 16])

D torch.Size([4, 10, 8])

💡 Hint

Remember batch_first=True means batch is the first dimension.

The RNN output has shape (batch_size, sequence_length, hidden_size) when batch_first=True. Here, batch_size=4, sequence_length=10, hidden_size=16.

❓ Hyperparameter

advanced

Choosing sequence length for training RNN text generation

Which sequence length is generally better for training an RNN text generator to capture long-term dependencies without causing too much memory use?

A Sequence length does not affect RNN training or memory.

B Very long sequences (e.g., 1000 characters) to capture all context at once.

C Moderate length sequences (e.g., 50-100 characters) balancing context and memory.

D Very short sequences (e.g., 5 characters) to speed up training.

💡 Hint

Think about the trade-off between context and computational resources.

Moderate sequence lengths allow the RNN to learn useful context without excessive memory use or vanishing gradients that happen with very long sequences.

❓ Metrics

advanced

Evaluating RNN text generation with perplexity

What does a lower perplexity score indicate when evaluating an RNN text generation model?

A The model is more confident and accurate in predicting the next character or word.

B The model is overfitting and memorizing the training data.

C The model has a higher loss value during training.

D The model is underfitting and not learning the data patterns.

💡 Hint

Perplexity measures how well the model predicts the next token.

Lower perplexity means the model assigns higher probabilities to the correct next tokens, showing better prediction performance.

🔧 Debug

expert

Identifying cause of exploding gradients in RNN training

During training an RNN for text generation, the loss suddenly becomes NaN and the model weights explode. Which code snippet is the most likely cause?

NLP

optimizer = torch.optim.Adam(model.parameters(), lr=1.0)
for inputs, targets in dataloader:
    optimizer.zero_grad()
    outputs = model(inputs)
    loss = loss_fn(outputs, targets)
    loss.backward()
    optimizer.step()

A The learning rate is too high (lr=1.0), causing unstable updates.

B The optimizer.zero_grad() is missing before loss.backward().

C The loss function is not differentiable, causing NaN gradients.

D The model inputs are not normalized, causing exploding gradients.

💡 Hint

Check the learning rate value and its effect on training stability.

A very high learning rate like 1.0 can cause large weight updates, leading to exploding gradients and NaN loss values.