Sequence padding in TensorFlow - Model Pipeline Trace

Model Pipeline - Sequence padding

This pipeline shows how sequence padding prepares text data of different lengths to be the same size so a model can learn from it easily.

Data Flow - 4 Stages

1Raw text sequences

5 sequences of varying lengths→Collect raw text sequences with different word counts→5 sequences of varying lengths

[['I', 'love', 'AI'], ['Hello'], ['Machine', 'learning', 'is', 'fun'], ['Data', 'science'], ['Hi', 'there']]

↓

2Tokenization

5 sequences of varying lengths→Convert words to numbers using a vocabulary→5 sequences of varying lengths (numbers)

[[2, 3, 4], [5], [6, 7, 8, 9], [10, 11], [12, 13]]

↓

3Sequence padding

5 sequences of varying lengths (numbers)→Add zeros at the end to make all sequences length 4→5 sequences of length 4

[[2, 3, 4, 0], [5, 0, 0, 0], [6, 7, 8, 9], [10, 11, 0, 0], [12, 13, 0, 0]]

↓

4Model input

5 sequences of length 4→Feed padded sequences into the model→Model input shape (5, 4)

Tensor with shape (5, 4) containing padded sequences

Training Trace - Epoch by Epoch

Loss
1.0 |          *
0.8 |        *  
0.6 |      *    
0.4 |    *      
0.2 |  *        
0.0 +----------
     1 2 3 4 5  Epochs

Epoch	Loss ↓	Accuracy ↑	Observation
1	0.85	0.50	Model starts learning with random weights
2	0.65	0.65	Loss decreases, accuracy improves as model learns patterns
3	0.50	0.75	Model continues to improve with training
4	0.40	0.82	Loss decreases steadily, accuracy rises
5	0.35	0.85	Training converges with good accuracy

Prediction Trace - 4 Layers

Layer 1: Input padded sequence

Layer 2: Embedding layer

Layer 3: LSTM layer

Layer 4: Dense output layer with softmax

Model Quiz - 3 Questions

Why do we add zeros to sequences during padding?

A To remove stop words from sequences

B To increase the vocabulary size

C To make all sequences the same length for the model

D To shuffle the sequence order

Key Insight

Sequence padding is essential to handle text data of different lengths by making them uniform. This allows models like LSTM to process batches efficiently and learn meaningful patterns, improving accuracy as training progresses.

Practice

1. What is the main purpose of sequence padding in TensorFlow?

easy

A. To make all sequences the same length by adding zeros

B. To shuffle the sequences randomly

C. To normalize the values in sequences

D. To remove zeros from sequences

Solution

1. Step 1: Understand sequence length requirements

   Models need inputs of the same length to process batches efficiently.

2. Step 2: Role of padding

   Padding adds zeros to shorter sequences to match the longest sequence length.

3. Final Answer:

   To make all sequences the same length by adding zeros -> Option A

4. Quick Check:

   Sequence padding = same length sequences [OK]

Hint: Padding adds zeros to equalize sequence lengths [OK]

Common Mistakes:

• Thinking padding shuffles sequences
• Confusing padding with normalization
• Assuming padding removes zeros

2. Which of the following is the correct way to pad sequences using TensorFlow's tf.keras.preprocessing.sequence.pad_sequences function?

easy

A. pad_sequences(sequences, maxlen=5, padding='post')

B. pad_sequences(sequences, maxlen=5, pad='after')

C. pad_sequences(sequences, length=5, padding='pre')

D. pad_sequences(sequences, max_length=5, padding='middle')

Solution

1. Step 1: Check function parameters

   The correct parameter for length is maxlen, and padding accepts 'pre' or 'post'.

2. Step 2: Validate option syntax

   pad_sequences(sequences, maxlen=5, padding='post') uses maxlen=5 and padding='post', which is correct syntax.

3. Final Answer:

   pad_sequences(sequences, maxlen=5, padding='post') -> Option A

4. Quick Check:

   Correct param names = pad_sequences(sequences, maxlen=5, padding='post') [OK]

Hint: Use maxlen and padding='pre' or 'post' in pad_sequences [OK]

Common Mistakes:

• Using wrong parameter names like pad or length
• Using invalid padding values like 'middle'
• Confusing maxlen with max_length

3. What will be the output of this code?

import tensorflow as tf
sequences = [[1, 2, 3], [4, 5], [6]]
padded = tf.keras.preprocessing.sequence.pad_sequences(sequences, maxlen=4, padding='pre')
print(padded.tolist())

medium

A. [[1, 2, 3], [4, 5], [6]]

B. [[0, 1, 2, 3], [0, 0, 4, 5], [0, 0, 0, 6]]

C. [[1, 2, 3, 0], [4, 5, 0, 0], [6, 0, 0, 0]]

D. [[0, 0, 1, 2], [0, 4, 5, 0], [6, 0, 0, 0]]

Solution

1. Step 1: Understand padding='pre'

   Padding adds zeros before the sequence to reach maxlen=4.

2. Step 2: Apply padding to each sequence

   First: [1,2,3] -> [0,1,2,3]; Second: [4,5] -> [0,0,4,5]; Third: [6] -> [0,0,0,6]

3. Final Answer:

   [[0, 1, 2, 3], [0, 0, 4, 5], [0, 0, 0, 6]] -> Option B

4. Quick Check:

   Pre-padding adds zeros before sequences [OK]

Hint: Pre-padding adds zeros at the start of sequences [OK]

Common Mistakes:

• Confusing pre-padding with post-padding
• Expecting original sequences without padding
• Miscounting zeros added

4. This code throws an error. What is the mistake?

import tensorflow as tf
sequences = [[1, 2], [3, 4, 5]]
padded = tf.keras.preprocessing.sequence.pad_sequences(sequences, maxlen=3, padding='middle')
print(padded)

medium

A. Sequences must be numpy arrays, not lists

B. maxlen should be larger than longest sequence

C. Invalid padding value 'middle' used

D. Missing argument 'value' for padding

Solution

1. Step 1: Check padding parameter values

   Valid padding values are 'pre' or 'post'. 'middle' is invalid.

2. Step 2: Identify error cause

   Using 'middle' causes a ValueError because it's not supported.

3. Final Answer:

   Invalid padding value 'middle' used -> Option C

4. Quick Check:

   Padding must be 'pre' or 'post' [OK]

Hint: Use only 'pre' or 'post' for padding parameter [OK]

Common Mistakes:

• Using unsupported padding values
• Thinking maxlen must be bigger than longest sequence
• Assuming lists cause errors

5. You have sequences of different lengths and want to pad them so that all sequences have length 6. You want zeros added after the sequences. Which code correctly achieves this and why?

hard

A. pad_sequences(sequences, maxlen=6, value=1) because it pads with ones to length 6

B. pad_sequences(sequences, maxlen=6, padding='pre') because it pads zeros before sequences to length 6

C. pad_sequences(sequences, maxlen=6, truncating='post') because it truncates sequences after length 6

D. pad_sequences(sequences, maxlen=6, padding='post') because it pads zeros after sequences to length 6

Solution

1. Step 1: Understand padding='post'

   Padding='post' adds zeros after the sequence to reach maxlen.

2. Step 2: Check other options

   pad_sequences(sequences, maxlen=6, padding='pre') because it pads zeros before sequences to length 6 pads before sequences, not after. pad_sequences(sequences, maxlen=6, truncating='post') because it truncates sequences after length 6 truncates sequences, not pads. pad_sequences(sequences, maxlen=6, value=1) because it pads with ones to length 6 pads with ones, not zeros.

3. Final Answer:

   pad_sequences(sequences, maxlen=6, padding='post') -> Option D

4. Quick Check:

   Post-padding adds zeros after sequences [OK]

Hint: Use padding='post' to add zeros after sequences [OK]

Common Mistakes:

• Confusing pre and post padding
• Using truncating instead of padding
• Changing pad value from zero unintentionally