Padding and sequence length in NLP - Model Pipeline Trace

Model Pipeline - Padding and sequence length

This pipeline shows how text sequences of different lengths are made the same length by adding padding. This helps the model learn from batches of data easily.

Data Flow - 4 Stages

1Raw text input

5 sequences of varying lengths→Collect sentences with different word counts→5 sequences of varying lengths

["I love AI", "Machine learning is fun", "Hello", "Deep learning", "NLP"]

↓

2Tokenization

5 sequences of varying lengths→Convert words to numbers (tokens)→5 sequences of varying lengths (token ids)

[[1, 2, 3], [4, 5, 6, 7], [8], [9, 10], [11]]

↓

3Padding sequences

5 sequences of varying lengths (token ids)→Add zeros to sequences to make all equal length (max length = 4)→5 sequences of length 4

[[1, 2, 3, 0], [4, 5, 6, 7], [8, 0, 0, 0], [9, 10, 0, 0], [11, 0, 0, 0]]

↓

4Model input

5 sequences of length 4→Feed padded sequences into the model→Model processes 5 sequences of length 4

Batch input shape: (5, 4)

Training Trace - Epoch by Epoch

Loss
1.0 |\
0.9 | \
0.8 |  \
0.7 |   \
0.6 |    \
0.5 |     \
0.4 |      \
0.3 |       \
    +----------------
     1 2 3 4 5 Epochs

Epoch	Loss ↓	Accuracy ↑	Observation
1	0.85	0.55	Model starts learning with padded sequences
2	0.65	0.70	Loss decreases, accuracy improves as model adapts
3	0.50	0.80	Model learns better representations with fixed-length input
4	0.40	0.85	Training converges with stable loss and high accuracy
5	0.35	0.88	Final epoch shows good performance on padded data

Prediction Trace - 4 Layers

Layer 1: Input padded sequence

Layer 2: Embedding layer

Layer 3: Recurrent layer

Layer 4: Output layer

Model Quiz - 3 Questions

Why do we add padding to sequences before feeding them to the model?

A To remove stop words from sequences

B To make all sequences the same length for batch processing

C To increase the vocabulary size

D To convert words into numbers

Key Insight

Padding sequences to the same length allows the model to process batches efficiently. This standardization helps the model learn better and converge faster during training.

Practice

1. What is the main purpose of padding in text sequences for machine learning models?

easy

A. To convert text into numbers without changing length

B. To make all sequences the same length by adding extra values

C. To randomly shuffle the words in sequences

D. To remove important words from sequences

Solution

1. Step 1: Understand padding concept

   Padding adds extra values (usually zeros) to sequences to make them all the same length.

2. Step 2: Recognize why padding is used

   This uniform length helps models process batches of data efficiently without errors.

3. Final Answer:

   To make all sequences the same length by adding extra values -> Option B

4. Quick Check:

   Padding = same length sequences [OK]

Hint: Padding adds extra tokens to equalize sequence lengths [OK]

Common Mistakes:

• Thinking padding removes words
• Confusing padding with shuffling
• Believing padding changes text meaning

2. Which of the following is the correct way to pad sequences using Python's Keras library?

easy

A. pad_sequences(sequences, maxlen=10, shuffle=True)

B. pad_sequences(sequences, maxlen=10, reverse=True)

C. pad_sequences(sequences, maxlen=10, padding='post')

D. pad_sequences(sequences, maxlen=10, drop=True)

Solution

1. Step 1: Identify correct padding function parameters

   Keras's pad_sequences uses 'padding' to specify where to add zeros, e.g., 'post' means after the sequence.

2. Step 2: Check options for valid parameters

   Only pad_sequences(sequences, maxlen=10, padding='post') uses a valid parameter 'padding' with a correct value 'post'. Others use invalid parameters like shuffle, reverse, drop.

3. Final Answer:

   pad_sequences(sequences, maxlen=10, padding='post') -> Option C

4. Quick Check:

   Correct padding param = pad_sequences(sequences, maxlen=10, padding='post') [OK]

Hint: Use 'padding' param in pad_sequences, not shuffle or drop [OK]

Common Mistakes:

• Using non-existent parameters like shuffle or drop
• Confusing padding location with sequence order
• Forgetting to set maxlen for fixed length

3. Given the code below, what will be the output shape of padded_sequences?

from tensorflow.keras.preprocessing.sequence import pad_sequences
sequences = [[1, 2, 3], [4, 5], [6]]
padded_sequences = pad_sequences(sequences, maxlen=4, padding='pre')

medium

A. (3, 4)

B. (4, 3)

C. (3, 3)

D. (4, 4)

Solution

1. Step 1: Count number of sequences

   There are 3 sequences: [1,2,3], [4,5], and [6].

2. Step 2: Understand padding effect on length

   maxlen=4 means each sequence is padded or truncated to length 4. So output shape is (3 sequences, 4 length each).

3. Final Answer:

   (3, 4) -> Option A

4. Quick Check:

   Number sequences = 3, length = 4 [OK]

Hint: Output shape = (number sequences, maxlen) [OK]

Common Mistakes:

• Confusing maxlen with number of sequences
• Mixing up padding='pre' with output shape
• Assuming shape changes with padding side

4. You wrote this code but get an error: TypeError: pad_sequences() got an unexpected keyword argument 'pad'. What is the likely mistake?

padded = pad_sequences(sequences, maxlen=5, pad='post')

medium

A. The parameter name should be 'padding', not 'pad'

B. maxlen must be smaller than sequence length

C. Sequences must be numpy arrays, not lists

D. pad_sequences requires a 'value' parameter

Solution

1. Step 1: Identify error cause from message

   The error says 'unexpected keyword argument pad', meaning 'pad' is not a valid parameter.

2. Step 2: Recall correct parameter name

   The correct parameter to specify padding side is 'padding', not 'pad'.

3. Final Answer:

   The parameter name should be 'padding', not 'pad' -> Option A

4. Quick Check:

   Correct param = 'padding' [OK]

Hint: Use 'padding' param, not 'pad' [OK]

Common Mistakes:

• Using 'pad' instead of 'padding'
• Assuming maxlen must be smaller than sequences
• Thinking sequences must be numpy arrays

5. You have text sequences of varying lengths. You want to pad them to length 10 but keep the last 10 words only if longer. Which code correctly achieves this using Keras?

hard

A. pad_sequences(sequences, maxlen=10, padding='post', truncating='pre')

B. pad_sequences(sequences, maxlen=10, padding='post', truncating='post')

C. pad_sequences(sequences, maxlen=10, padding='pre', truncating='post')

D. pad_sequences(sequences, maxlen=10, padding='pre', truncating='pre')

Solution

1. Step 1: Understand padding and truncating sides

   Padding='pre' adds zeros at the start; truncating='pre' removes words from the start, keeping last words.

2. Step 2: Match requirement to keep last 10 words

   To keep last 10 words, truncate from the start ('pre') and pad at the start ('pre').

3. Final Answer:

   pad_sequences(sequences, maxlen=10, padding='pre', truncating='pre') -> Option D

4. Quick Check:

   Keep last words = truncating='pre' [OK]

Hint: Use truncating='pre' to keep last words, padding='pre' to pad start [OK]

Common Mistakes:

• Using padding='post' which pads end instead of start
• Using truncating='post' which removes last words
• Confusing padding and truncating parameters