Text preprocessing for RNNs in PyTorch - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of tokenization and padding

Given the following PyTorch code for tokenizing and padding sequences for an RNN, what is the output of the padded tensor?

PyTorch

import torch
from torch.nn.utils.rnn import pad_sequence

sequences = [torch.tensor([1, 2, 3]), torch.tensor([4, 5]), torch.tensor([6])]
padded = pad_sequence(sequences, batch_first=True, padding_value=0)
print(padded)

A tensor([[1, 2, 3], [4, 5, 6], [0, 0, 0]])

B tensor([[1, 2, 3], [4, 5, 0], [6, 0, 0]])

C tensor([[1, 2, 3], [0, 4, 5], [0, 0, 6]])

D tensor([[1, 2, 3], [4, 5], [6]])

💡 Hint

Remember pad_sequence aligns sequences by padding shorter ones with zeros at the end when batch_first=True.

pad_sequence with batch_first=True pads shorter sequences with the padding_value (0) at the end, so all sequences have the same length as the longest one (3 here).

🧠 Conceptual

intermediate

Why use padding in RNN input sequences?

Why do we pad sequences to the same length before feeding them into an RNN?

A Because RNNs require inputs of uniform length to process batches efficiently.

B Because padding improves the accuracy of the RNN model.

C Because padding converts text into numerical vectors.

D Because padding removes stop words from the sequences.

💡 Hint

Think about how batches are processed in parallel.

RNNs process batches of sequences in parallel, which requires all sequences in a batch to have the same length. Padding shorter sequences ensures uniform length.

❓ Hyperparameter

advanced

Choosing max sequence length for padding

When preprocessing text for an RNN, what is a common approach to decide the maximum sequence length for padding?

A Set max length to the length of the longest sequence in the entire dataset.

B Set max length to the average length of all sequences without considering outliers.

C Set max length to a fixed number based on domain knowledge or percentile of sequence lengths.

D Set max length to the length of the shortest sequence to save memory.

💡 Hint

Consider balancing memory use and information retention.

Choosing a fixed max length based on domain knowledge or a percentile (e.g., 95th percentile) balances keeping most data and limiting padding overhead.

🔧 Debug

advanced

Error in embedding input shape for RNN

What error will this PyTorch code raise when feeding input to an RNN embedding layer? import torch import torch.nn as nn embedding = nn.Embedding(10, 3) inputs = torch.tensor([1, 2, 3, 4]) embedded = embedding(inputs) rnn = nn.RNN(input_size=3, hidden_size=5, batch_first=True) output, hidden = rnn(embedded)

A RuntimeError: Expected 3D input for RNN but got 2D input

B TypeError: embedding() missing 1 required positional argument

C IndexError: index out of range in embedding

D No error, code runs successfully

💡 Hint

Check the shape of the tensor passed to the RNN.

The RNN expects a 3D input tensor (batch_size, seq_len, input_size), but embedded has shape (seq_len, embedding_dim) which is 2D, causing a RuntimeError.

❓ Model Choice

expert

Best preprocessing for variable-length text sequences in RNN training

You have a dataset of sentences with widely varying lengths. You want to train an RNN efficiently. Which preprocessing approach is best?

A Convert all sequences to one-hot vectors without padding.

B Truncate all sequences to the shortest sentence length to avoid padding.

C Pad all sequences to the length of the longest sentence in the dataset.

D Pad sequences to a fixed max length and use PyTorch's pack_padded_sequence to handle variable lengths.

💡 Hint

Consider both efficiency and preserving sequence information.

Using pack_padded_sequence with padded sequences allows the RNN to ignore padded tokens and process variable-length sequences efficiently.