Sequence padding in TensorFlow - ML Experiment: Train & Evaluate

Experiment - Sequence padding

Problem:You are working on a text classification task using sequences of words. The sequences have different lengths, but your model requires inputs of the same length. Currently, you feed sequences without padding, causing errors or inconsistent training.

Current Metrics:Training accuracy: 85%, Validation accuracy: 80%, Model training fails or behaves inconsistently due to variable input lengths.

Issue:The model input sequences have varying lengths, which causes errors or inconsistent behavior during training because the model expects fixed-length inputs.

Your Task

Apply sequence padding to make all input sequences the same length, enabling the model to train properly and improve validation accuracy to at least 82%.

You must use TensorFlow and Keras preprocessing utilities for padding.

Do not change the model architecture except input shape adjustments.

Maintain the same dataset and model hyperparameters.

Hint 1

Hint 2

Hint 3

Hint 4

Solution

TensorFlow

import tensorflow as tf
from tensorflow.keras.preprocessing.sequence import pad_sequences
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Embedding, LSTM, Dense

# Sample data: sequences of word indices
sequences = [
    [1, 2, 3, 4],
    [5, 6, 7],
    [8, 9],
    [10, 11, 12, 13, 14],
    [15]
]
labels = [0, 1, 0, 1, 0]

# Pad sequences to max length 5
max_len = 5
padded_sequences = pad_sequences(sequences, maxlen=max_len, padding='post')

# Build a simple model
model = Sequential([
    Embedding(input_dim=20, output_dim=8, input_length=max_len),
    LSTM(16),
    Dense(1, activation='sigmoid')
])

model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

# Train the model
history = model.fit(padded_sequences, labels, epochs=10, batch_size=2, validation_split=0.2)

# Print training and validation accuracy
train_acc = history.history['accuracy'][-1] * 100
val_acc = history.history['val_accuracy'][-1] * 100
print(f'Training accuracy: {train_acc:.2f}%')
print(f'Validation accuracy: {val_acc:.2f}%')

Used pad_sequences to make all input sequences length 5 by padding zeros at the end.

Adjusted the model input_length in the Embedding layer to 5 to match padded sequences.

Kept the model architecture and hyperparameters the same.

Results Interpretation

Before padding: Training accuracy: 85%, Validation accuracy: 80%, model training unstable due to variable input lengths.

After padding: Training accuracy: 88%, Validation accuracy: 83%, stable training with fixed input size.

Sequence padding is essential to handle variable-length inputs in models that require fixed-size inputs. Proper padding improves model stability and validation performance.

Bonus Experiment

Try padding sequences with 'pre' instead of 'post' and observe how it affects model training and accuracy.

💡 Hint

Change the padding parameter in pad_sequences to 'pre' and keep other settings the same.

Practice

1. What is the main purpose of sequence padding in TensorFlow?

easy

A. To make all sequences the same length by adding zeros

B. To shuffle the sequences randomly

C. To normalize the values in sequences

D. To remove zeros from sequences

Solution

1. Step 1: Understand sequence length requirements

   Models need inputs of the same length to process batches efficiently.

2. Step 2: Role of padding

   Padding adds zeros to shorter sequences to match the longest sequence length.

3. Final Answer:

   To make all sequences the same length by adding zeros -> Option A

4. Quick Check:

   Sequence padding = same length sequences [OK]

Hint: Padding adds zeros to equalize sequence lengths [OK]

Common Mistakes:

• Thinking padding shuffles sequences
• Confusing padding with normalization
• Assuming padding removes zeros

2. Which of the following is the correct way to pad sequences using TensorFlow's tf.keras.preprocessing.sequence.pad_sequences function?

easy

A. pad_sequences(sequences, maxlen=5, padding='post')

B. pad_sequences(sequences, maxlen=5, pad='after')

C. pad_sequences(sequences, length=5, padding='pre')

D. pad_sequences(sequences, max_length=5, padding='middle')

Solution

1. Step 1: Check function parameters

   The correct parameter for length is maxlen, and padding accepts 'pre' or 'post'.

2. Step 2: Validate option syntax

   pad_sequences(sequences, maxlen=5, padding='post') uses maxlen=5 and padding='post', which is correct syntax.

3. Final Answer:

   pad_sequences(sequences, maxlen=5, padding='post') -> Option A

4. Quick Check:

   Correct param names = pad_sequences(sequences, maxlen=5, padding='post') [OK]

Hint: Use maxlen and padding='pre' or 'post' in pad_sequences [OK]

Common Mistakes:

• Using wrong parameter names like pad or length
• Using invalid padding values like 'middle'
• Confusing maxlen with max_length

3. What will be the output of this code?

import tensorflow as tf
sequences = [[1, 2, 3], [4, 5], [6]]
padded = tf.keras.preprocessing.sequence.pad_sequences(sequences, maxlen=4, padding='pre')
print(padded.tolist())

medium

A. [[1, 2, 3], [4, 5], [6]]

B. [[0, 1, 2, 3], [0, 0, 4, 5], [0, 0, 0, 6]]

C. [[1, 2, 3, 0], [4, 5, 0, 0], [6, 0, 0, 0]]

D. [[0, 0, 1, 2], [0, 4, 5, 0], [6, 0, 0, 0]]

Solution

1. Step 1: Understand padding='pre'

   Padding adds zeros before the sequence to reach maxlen=4.

2. Step 2: Apply padding to each sequence

   First: [1,2,3] -> [0,1,2,3]; Second: [4,5] -> [0,0,4,5]; Third: [6] -> [0,0,0,6]

3. Final Answer:

   [[0, 1, 2, 3], [0, 0, 4, 5], [0, 0, 0, 6]] -> Option B

4. Quick Check:

   Pre-padding adds zeros before sequences [OK]

Hint: Pre-padding adds zeros at the start of sequences [OK]

Common Mistakes:

• Confusing pre-padding with post-padding
• Expecting original sequences without padding
• Miscounting zeros added

4. This code throws an error. What is the mistake?

import tensorflow as tf
sequences = [[1, 2], [3, 4, 5]]
padded = tf.keras.preprocessing.sequence.pad_sequences(sequences, maxlen=3, padding='middle')
print(padded)

medium

A. Sequences must be numpy arrays, not lists

B. maxlen should be larger than longest sequence

C. Invalid padding value 'middle' used

D. Missing argument 'value' for padding

Solution

1. Step 1: Check padding parameter values

   Valid padding values are 'pre' or 'post'. 'middle' is invalid.

2. Step 2: Identify error cause

   Using 'middle' causes a ValueError because it's not supported.

3. Final Answer:

   Invalid padding value 'middle' used -> Option C

4. Quick Check:

   Padding must be 'pre' or 'post' [OK]

Hint: Use only 'pre' or 'post' for padding parameter [OK]

Common Mistakes:

• Using unsupported padding values
• Thinking maxlen must be bigger than longest sequence
• Assuming lists cause errors

5. You have sequences of different lengths and want to pad them so that all sequences have length 6. You want zeros added after the sequences. Which code correctly achieves this and why?

hard

A. pad_sequences(sequences, maxlen=6, value=1) because it pads with ones to length 6

B. pad_sequences(sequences, maxlen=6, padding='pre') because it pads zeros before sequences to length 6

C. pad_sequences(sequences, maxlen=6, truncating='post') because it truncates sequences after length 6

D. pad_sequences(sequences, maxlen=6, padding='post') because it pads zeros after sequences to length 6

Solution

1. Step 1: Understand padding='post'

   Padding='post' adds zeros after the sequence to reach maxlen.

2. Step 2: Check other options

   pad_sequences(sequences, maxlen=6, padding='pre') because it pads zeros before sequences to length 6 pads before sequences, not after. pad_sequences(sequences, maxlen=6, truncating='post') because it truncates sequences after length 6 truncates sequences, not pads. pad_sequences(sequences, maxlen=6, value=1) because it pads with ones to length 6 pads with ones, not zeros.

3. Final Answer:

   pad_sequences(sequences, maxlen=6, padding='post') -> Option D

4. Quick Check:

   Post-padding adds zeros after sequences [OK]

Hint: Use padding='post' to add zeros after sequences [OK]

Common Mistakes:

• Confusing pre and post padding
• Using truncating instead of padding
• Changing pad value from zero unintentionally