Text preprocessing for RNNs in PyTorch - Deep Dive

Overview - Text preprocessing for RNNs

What is it?

Text preprocessing for RNNs means preparing raw text data so that a Recurrent Neural Network (RNN) can understand and learn from it. This involves turning words or characters into numbers, organizing sequences, and making sure all inputs have the same length. Without this step, the RNN cannot process text because it only works with numbers in fixed-size batches.

Why it matters

Text data is messy and varies in length and format. Without preprocessing, RNNs would get confused by different sentence lengths and unknown words. Proper preprocessing makes training faster, more stable, and helps the model learn meaningful patterns. Without it, language models would perform poorly or fail to learn at all.

Where it fits

Before this, learners should understand basic Python programming and how neural networks work. After mastering text preprocessing, learners can move on to building and training RNN models, then explore advanced topics like attention mechanisms or transformers.

Mental Model

Core Idea

Text preprocessing transforms messy, variable-length text into clean, fixed-size numeric sequences that RNNs can process efficiently.

Think of it like...

It's like preparing ingredients before cooking: chopping vegetables into uniform pieces so they cook evenly and mix well in the recipe.

Raw Text → Tokenization → Vocabulary Mapping → Sequence Padding → Numeric Tensor Input

┌─────────┐    ┌─────────────┐    ┌───────────────┐    ┌─────────────┐    ┌───────────────┐
│  Raw    │ → │ Tokenizer   │ → │ Vocabulary    │ → │ Padding     │ → │ Numeric Input │
│  Text   │    │ (split text)│    │ (word to idx) │    │ (fix length)│    │ (tensor)      │
└─────────┘    └─────────────┘    └───────────────┘    └─────────────┘    └───────────────┘

Build-Up - 7 Steps

1

Foundation

Understanding Raw Text Data

Concept: Raw text is a sequence of characters or words that computers cannot directly use for math operations.

Text data looks like sentences or paragraphs made of letters and spaces. Computers need numbers, so we must convert text into numbers before feeding it to an RNN. This step is the very first in preprocessing.

Result

You realize raw text cannot be input directly into neural networks.

Understanding that text is not numeric explains why preprocessing is necessary before any machine learning.

2

Foundation

Tokenization: Splitting Text into Pieces

Concept: Tokenization breaks text into smaller units called tokens, usually words or characters.

For example, the sentence 'I love AI' becomes ['I', 'love', 'AI'] when tokenized by words. Tokenization helps us handle text piece by piece and assign numbers to each token.

Result

Text is now a list of tokens, easier to map to numbers.

Tokenization is the bridge from raw text to structured data that can be numerically encoded.

3

Intermediate

Building Vocabulary and Mapping Tokens

🤔Before reading on: do you think each unique word should have a unique number or can multiple words share the same number? Commit to your answer.

Concept: Vocabulary is a list of all unique tokens, each assigned a unique number (index).

We collect all tokens from the dataset and assign each a unique integer ID. For example, 'I' → 1, 'love' → 2, 'AI' → 3. Unknown words get a special token like ''. This mapping lets us convert token lists into number lists.

Result

Tokens can be replaced by their numeric IDs, creating sequences of numbers.

Knowing that each token maps to a unique number is key to converting text into a format RNNs can process.

4

Intermediate

Sequence Padding and Truncation

🤔Before reading on: do you think all input sequences must be the same length for RNNs, or can they vary freely? Commit to your answer.

Concept: RNNs require input sequences in batches to have the same length, so shorter sequences are padded and longer ones truncated.

We decide a fixed length (e.g., 10 tokens). Sequences shorter than 10 get special padding tokens '' added at the end. Longer sequences are cut off after 10 tokens. This makes batch processing possible and efficient.

Result

All sequences have uniform length, enabling batch training.

Understanding padding prevents errors and inefficiencies during model training.

5

Intermediate

Converting Sequences to PyTorch Tensors

🤔Before reading on: do you think PyTorch models accept Python lists directly or require tensors? Commit to your answer.

Concept: PyTorch models require input data as tensors, which are multi-dimensional arrays optimized for computation.

After padding, sequences of numbers are converted into PyTorch tensors using torch.tensor(). These tensors can be moved to GPUs and used in RNN models.

Result

Data is ready for efficient processing by PyTorch RNNs.

Knowing the tensor format is essential for using PyTorch models correctly.

6

Advanced

Handling Unknown and Rare Words

🤔Before reading on: do you think every word in test data will appear in training vocabulary? Commit to your answer.

Concept: Unknown or rare words are replaced with a special token to handle words not seen during training.

When new text contains words outside the vocabulary, we replace them with ''. This prevents errors and helps the model generalize. Rare words can also be grouped under '' to reduce vocabulary size.

Result

Model can handle new or rare words gracefully during inference.

Handling unknown words is crucial for real-world robustness of text models.

7

Expert

Optimizing Preprocessing for Variable Lengths

🤔Before reading on: do you think padding all sequences to the max length in the dataset is always best? Commit to your answer.

Concept: Advanced techniques like packing padded sequences improve efficiency by telling RNNs actual sequence lengths.

Instead of padding all sequences to the longest one, PyTorch provides utilities like pack_padded_sequence that let RNNs ignore padding tokens during computation. This speeds up training and reduces wasted computation.

Result

Training becomes faster and more memory-efficient without losing information.

Knowing how to use packed sequences unlocks better performance in production RNN models.

Under the Hood

Text preprocessing converts strings into numeric tensors by tokenizing text, mapping tokens to indices, and padding sequences to uniform length. Internally, RNNs process these tensors step-by-step over time. Padding tokens are usually masked or ignored during training to prevent them from affecting learning. Special tokens like '' and '' are reserved indices in the vocabulary. PyTorch tensors store these sequences efficiently in memory and enable GPU acceleration.

Why designed this way?

RNNs require fixed-size numeric inputs for batch processing and matrix operations. Variable-length text sequences would break batch computations and slow training. Padding and token mapping standardize inputs, while special tokens handle edge cases like unknown words. This design balances flexibility with computational efficiency, enabling scalable training on large text datasets.

Raw Text
   ↓ Tokenization
Tokens List
   ↓ Vocabulary Mapping
Numeric Sequences
   ↓ Padding/Truncation
Fixed-Length Sequences
   ↓ PyTorch Tensor Conversion
Tensor Input → RNN Model

[Special Tokens: <PAD>=0, <UNK>=1]

Batch Processing:
┌───────────────┐
│ Sequence 1    │
│ [5, 7, 2, 0]  │
│ Sequence 2    │
│ [3, 9, 0, 0]  │
└───────────────┘

Padding tokens (0) ignored during training via masking or packing.

Myth Busters - 4 Common Misconceptions

Quick: Do you think RNNs can handle raw text strings directly as input? Commit to yes or no.

Common Belief: RNNs can take raw text strings as input and learn from them directly.

Reality: RNNs require numeric tensors as input; raw text must be converted into numbers first.

Why it matters: Trying to feed raw text causes errors and prevents model training.

Quick: Do you think padding sequences with zeros changes the meaning of the text? Commit to yes or no.

Common Belief: Padding sequences with zeros adds meaningful data that affects model predictions.

Reality: Padding tokens are placeholders with no meaning and are ignored or masked during training.

Why it matters: Misunderstanding padding can lead to incorrect model evaluation or training bugs.

Quick: Do you think every word in test data will always be in the training vocabulary? Commit to yes or no.

Common Belief: All words in new text will be known from training vocabulary, so no special handling is needed.

Reality: New or rare words often appear and must be replaced with an unknown token.

Why it matters: Ignoring unknown words causes errors or poor model generalization on real data.

Quick: Do you think padding all sequences to the longest sequence length is always the best approach? Commit to yes or no.

Common Belief: Padding all sequences to the maximum length in the dataset is always optimal.

Reality: Padding to the max length wastes computation; packing sequences is more efficient.

Why it matters: Not using packed sequences slows training and wastes memory, especially with very long sequences.

Expert Zone

1

Vocabulary size impacts model size and training speed; balancing coverage and size is key.

2

Choice of tokenization (word vs. subword vs. character) affects model ability to handle rare words and generalize.

3

Using packed sequences requires careful tracking of original sequence lengths and sorting batches by length.

When NOT to use

For very long texts or documents, RNNs with simple padding become inefficient; transformers or CNNs with attention mechanisms are better alternatives. Also, for languages with complex morphology, subword tokenization or byte-pair encoding may be preferred over simple word tokenization.

Production Patterns

In production, preprocessing pipelines often include caching vocabularies, using subword tokenizers like SentencePiece, and applying packed sequences for efficient batch training. Real systems also handle streaming text by incremental tokenization and dynamic padding.

Connections

One-hot Encoding

Text preprocessing builds on the idea of representing categorical data as numbers, similar to one-hot encoding.

Understanding one-hot encoding helps grasp how tokens can be represented as vectors before embedding layers.

Signal Processing

Both text preprocessing and signal processing involve converting raw signals into fixed-size numeric sequences for analysis.

Recognizing this connection shows how different fields solve the problem of variable-length input data.

Human Language Learning

Preprocessing mimics how humans break down language into words and meanings before understanding context.

Knowing this helps appreciate why tokenization and vocabulary building are natural steps in language modeling.

Common Pitfalls

#1 Feeding raw text strings directly into the RNN model.

Wrong approach:

model(input_text) # input_text is a list of strings like ['I love AI']

Correct approach:

model(input_tensor) # input_tensor is a padded tensor of token indices

Root cause:

Misunderstanding that models require numeric tensor inputs, not raw strings.

#2 Not padding sequences to the same length before batching.

Wrong approach:

batch = torch.tensor([[1,2,3], [4,5]]) # sequences of different lengths without padding

Correct approach:

batch = torch.tensor([[1,2,3], [4,5,0]]) # padded sequences with token

Root cause:

Not knowing that batch tensors must have uniform dimensions for processing.

#3 Ignoring unknown words during inference, causing errors.

Wrong approach:

token_id = vocab[word] # raises KeyError if word not in vocab

Correct approach:

token_id = vocab.get(word, vocab['']) # safely maps unknown words

Root cause:

Assuming test data words always appear in training vocabulary.

Key Takeaways

Text preprocessing converts raw text into fixed-length numeric sequences that RNNs can process.

Tokenization splits text into manageable pieces, and vocabulary mapping assigns each token a unique number.

Padding sequences to the same length enables efficient batch processing but requires masking or packing to avoid learning from padding.

Handling unknown words with special tokens ensures models can generalize to new data.

Advanced techniques like packed sequences improve training efficiency by ignoring padding during computation.

Practice

1. Why do we split text into tokens before feeding it to an RNN?

easy

A. Because RNNs process sequences of numbers, not raw text

B. To reduce the size of the dataset

C. To make the text look nicer

D. Because tokens are easier to print

Solution

1. Step 1: Understand RNN input requirements

   RNNs work with sequences of numbers, not raw text strings.

2. Step 2: Role of tokenization

   Splitting text into tokens converts sentences into smaller units that can be mapped to numbers.

3. Final Answer:

   Because RNNs process sequences of numbers, not raw text -> Option A

4. Quick Check:

   Tokenization = Convert text to numbers [OK]

Hint: RNNs need numbers, so split text into tokens first [OK]

Common Mistakes:

• Thinking tokens are for making text prettier
• Believing tokenization reduces dataset size
• Confusing tokens with characters

2. Which PyTorch function is commonly used to pad sequences to the same length for batch processing?

easy

A. torch.nn.utils.rnn.pad_sequence

B. torch.tensor.pad

C. torch.pad_sequences

D. torch.nn.pad

Solution

1. Step 1: Identify PyTorch padding utilities

   PyTorch provides pad_sequence in torch.nn.utils.rnn to pad variable-length sequences.

2. Step 2: Check other options

   Functions like torch.tensor.pad or torch.nn.pad do not exist; torch.pad_sequences is not a PyTorch function.

3. Final Answer:

   torch.nn.utils.rnn.pad_sequence -> Option A

4. Quick Check:

   Use pad_sequence to pad RNN inputs [OK]

Hint: Remember: pad_sequence is in torch.nn.utils.rnn [OK]

Common Mistakes:

• Using non-existent torch.pad_sequences
• Confusing tensor.pad with pad_sequence
• Trying to pad manually without this function

3. Given the following code, what is the shape of the padded batch tensor?

import torch
from torch.nn.utils.rnn import pad_sequence

seq1 = torch.tensor([1, 2, 3])
seq2 = torch.tensor([4, 5])
seq3 = torch.tensor([6])
batch = pad_sequence([seq1, seq2, seq3], batch_first=True, padding_value=0)
print(batch.shape)

medium

A. (1, 3)

B. (3, 1)

C. (3, 3)

D. (3, 6)

Solution

1. Step 1: Understand input sequences

   Sequences have lengths 3, 2, and 1 respectively.

2. Step 2: pad_sequence with batch_first=true

   All sequences are padded to length 3 (max length), batch dimension is first, so shape is (3 sequences, 3 elements each).

3. Final Answer:

   (3, 3) -> Option C

4. Quick Check:

   Batch size = 3, max seq length = 3 [OK]

Hint: Batch shape = (number sequences, max sequence length) [OK]

Common Mistakes:

• Confusing batch_first=true with false
• Assuming padding adds length beyond max sequence
• Mixing up batch and sequence dimensions

4. What is wrong with this code snippet for preparing text sequences for an RNN?

import torch
from torch.nn.utils.rnn import pad_sequence

sentences = [[1, 2, 3, 4], [5, 6], [7]]
tensors = [torch.tensor(s) for s in sentences]
padded = pad_sequence(tensors)
print(padded.shape)

medium

A. torch.tensor cannot convert lists to tensors

B. pad_sequence is missing batch_first=true, so shape is unexpected

C. pad_sequence requires padding_value argument

D. The input lists must be numpy arrays, not lists

Solution

1. Step 1: Check pad_sequence default behavior

   By default, pad_sequence returns tensor with shape (max_seq_len, batch_size), not batch first.

2. Step 2: Effect on output shape

   Without batch_first=true, the printed shape will be (4, 3) instead of expected batch-first (3, 4) shape.

3. Final Answer:

   pad_sequence is missing batch_first=true, so shape is unexpected -> Option B

4. Quick Check:

   Use batch_first=true for (batch, seq_len) shape [OK]

Hint: Always add batch_first=true for batch as first dimension [OK]

Common Mistakes:

• Assuming pad_sequence pads automatically without batch_first
• Thinking torch.tensor can't convert lists
• Believing padding_value is mandatory

5. You have a batch of sentences tokenized as integer lists of different lengths. You want to feed them into an RNN in PyTorch. Which sequence of steps is correct for preprocessing?

hard

A. Tokenize text -> Pad sequences -> Convert tokens to integers -> Feed to RNN

B. Pad raw text strings -> Tokenize padded strings -> Convert tokens to integers -> Feed to RNN

C. Convert raw text to tensor -> Tokenize tensor -> Pad sequences -> Feed to RNN

D. Tokenize text -> Convert tokens to integers -> Pad sequences with pad_sequence(batch_first=true) -> Convert to tensor batch

Solution

1. Step 1: Tokenize text and convert tokens to integers

   First, split text into tokens, then map tokens to integers using a vocabulary.

2. Step 2: Pad sequences and prepare batch tensor

   Pad integer sequences to equal length using pad_sequence with batch_first=true, then feed the tensor batch to the RNN.

3. Final Answer:

   Tokenize text -> Convert tokens to integers -> Pad sequences with pad_sequence(batch_first=true) -> Convert to tensor batch -> Option D

4. Quick Check:

   Tokenize -> Integer map -> Pad -> Batch tensor [OK]

Hint: Tokenize first, then integer map, then pad sequences [OK]

Common Mistakes:

• Padding raw text instead of token integers
• Converting raw text directly to tensor
• Padding before converting tokens to integers