Sequence padding in TensorFlow

Introduction

Sequence padding makes all sequences the same length so a model can process them together easily.

When you have sentences of different lengths and want to train a text classifier.

When processing time series data with varying lengths in a batch.

When feeding sequences into a neural network that requires fixed input size.

When preparing data for models like RNNs or Transformers that expect uniform input shapes.

Syntax

TensorFlow

tf.keras.preprocessing.sequence.pad_sequences(sequences, maxlen=None, dtype='int32', padding='pre', truncating='pre', value=0)

sequences: List of lists with varying lengths.

padding: 'pre' adds zeros at the start, 'post' adds zeros at the end.

Examples

Default padding adds zeros at the start to make all sequences length 3.

TensorFlow

import tensorflow as tf
sequences = [[1, 2, 3], [4, 5], [6]]
padded = tf.keras.preprocessing.sequence.pad_sequences(sequences)
print(padded)

Padding with 'post' adds zeros at the end instead of the start.

TensorFlow

padded_post = tf.keras.preprocessing.sequence.pad_sequences(sequences, padding='post')
print(padded_post)

Truncates sequences longer than 2 and pads shorter ones to length 2.

TensorFlow

padded_maxlen = tf.keras.preprocessing.sequence.pad_sequences(sequences, maxlen=2)
print(padded_maxlen)

Sample Model

This code pads three sequences to length 4 by adding zeros at the end. It prints the padded sequences and their shape.

TensorFlow

import tensorflow as tf

# Sample sequences of different lengths
sequences = [[7, 8, 9], [10, 11], [12]]

# Pad sequences to length 4 with zeros at the end
padded_sequences = tf.keras.preprocessing.sequence.pad_sequences(sequences, maxlen=4, padding='post')

print('Padded sequences:')
print(padded_sequences)

# Check shape to confirm uniform length
print('Shape:', padded_sequences.shape)

Important Notes

Padding is important because models expect inputs of the same size.

Choose 'pre' or 'post' padding based on what fits your model best.

Use maxlen to control the final sequence length, truncating longer sequences if needed.

Summary

Sequence padding makes sequences the same length by adding zeros.

It helps models process batches of sequences together.

Padding can be added before or after the sequence.

Practice

1. What is the main purpose of sequence padding in TensorFlow?

easy

A. To make all sequences the same length by adding zeros

B. To shuffle the sequences randomly

C. To normalize the values in sequences

D. To remove zeros from sequences

Solution

1. Step 1: Understand sequence length requirements

   Models need inputs of the same length to process batches efficiently.

2. Step 2: Role of padding

   Padding adds zeros to shorter sequences to match the longest sequence length.

3. Final Answer:

   To make all sequences the same length by adding zeros -> Option A

4. Quick Check:

   Sequence padding = same length sequences [OK]

Hint: Padding adds zeros to equalize sequence lengths [OK]

Common Mistakes:

• Thinking padding shuffles sequences
• Confusing padding with normalization
• Assuming padding removes zeros

2. Which of the following is the correct way to pad sequences using TensorFlow's tf.keras.preprocessing.sequence.pad_sequences function?

easy

A. pad_sequences(sequences, maxlen=5, padding='post')

B. pad_sequences(sequences, maxlen=5, pad='after')

C. pad_sequences(sequences, length=5, padding='pre')

D. pad_sequences(sequences, max_length=5, padding='middle')

Solution

1. Step 1: Check function parameters

   The correct parameter for length is maxlen, and padding accepts 'pre' or 'post'.

2. Step 2: Validate option syntax

   pad_sequences(sequences, maxlen=5, padding='post') uses maxlen=5 and padding='post', which is correct syntax.

3. Final Answer:

   pad_sequences(sequences, maxlen=5, padding='post') -> Option A

4. Quick Check:

   Correct param names = pad_sequences(sequences, maxlen=5, padding='post') [OK]

Hint: Use maxlen and padding='pre' or 'post' in pad_sequences [OK]

Common Mistakes:

• Using wrong parameter names like pad or length
• Using invalid padding values like 'middle'
• Confusing maxlen with max_length

3. What will be the output of this code?

import tensorflow as tf
sequences = [[1, 2, 3], [4, 5], [6]]
padded = tf.keras.preprocessing.sequence.pad_sequences(sequences, maxlen=4, padding='pre')
print(padded.tolist())

medium

A. [[1, 2, 3], [4, 5], [6]]

B. [[0, 1, 2, 3], [0, 0, 4, 5], [0, 0, 0, 6]]

C. [[1, 2, 3, 0], [4, 5, 0, 0], [6, 0, 0, 0]]

D. [[0, 0, 1, 2], [0, 4, 5, 0], [6, 0, 0, 0]]

Solution

1. Step 1: Understand padding='pre'

   Padding adds zeros before the sequence to reach maxlen=4.

2. Step 2: Apply padding to each sequence

   First: [1,2,3] -> [0,1,2,3]; Second: [4,5] -> [0,0,4,5]; Third: [6] -> [0,0,0,6]

3. Final Answer:

   [[0, 1, 2, 3], [0, 0, 4, 5], [0, 0, 0, 6]] -> Option B

4. Quick Check:

   Pre-padding adds zeros before sequences [OK]

Hint: Pre-padding adds zeros at the start of sequences [OK]

Common Mistakes:

• Confusing pre-padding with post-padding
• Expecting original sequences without padding
• Miscounting zeros added

4. This code throws an error. What is the mistake?

import tensorflow as tf
sequences = [[1, 2], [3, 4, 5]]
padded = tf.keras.preprocessing.sequence.pad_sequences(sequences, maxlen=3, padding='middle')
print(padded)

medium

A. Sequences must be numpy arrays, not lists

B. maxlen should be larger than longest sequence

C. Invalid padding value 'middle' used

D. Missing argument 'value' for padding

Solution

1. Step 1: Check padding parameter values

   Valid padding values are 'pre' or 'post'. 'middle' is invalid.

2. Step 2: Identify error cause

   Using 'middle' causes a ValueError because it's not supported.

3. Final Answer:

   Invalid padding value 'middle' used -> Option C

4. Quick Check:

   Padding must be 'pre' or 'post' [OK]

Hint: Use only 'pre' or 'post' for padding parameter [OK]

Common Mistakes:

• Using unsupported padding values
• Thinking maxlen must be bigger than longest sequence
• Assuming lists cause errors

5. You have sequences of different lengths and want to pad them so that all sequences have length 6. You want zeros added after the sequences. Which code correctly achieves this and why?

hard

A. pad_sequences(sequences, maxlen=6, value=1) because it pads with ones to length 6

B. pad_sequences(sequences, maxlen=6, padding='pre') because it pads zeros before sequences to length 6

C. pad_sequences(sequences, maxlen=6, truncating='post') because it truncates sequences after length 6

D. pad_sequences(sequences, maxlen=6, padding='post') because it pads zeros after sequences to length 6

Solution

1. Step 1: Understand padding='post'

   Padding='post' adds zeros after the sequence to reach maxlen.

2. Step 2: Check other options

   pad_sequences(sequences, maxlen=6, padding='pre') because it pads zeros before sequences to length 6 pads before sequences, not after. pad_sequences(sequences, maxlen=6, truncating='post') because it truncates sequences after length 6 truncates sequences, not pads. pad_sequences(sequences, maxlen=6, value=1) because it pads with ones to length 6 pads with ones, not zeros.

3. Final Answer:

   pad_sequences(sequences, maxlen=6, padding='post') -> Option D

4. Quick Check:

   Post-padding adds zeros after sequences [OK]

Hint: Use padding='post' to add zeros after sequences [OK]

Common Mistakes:

• Confusing pre and post padding
• Using truncating instead of padding
• Changing pad value from zero unintentionally