Padding and stride in TensorFlow - ML Experiment: Train & Evaluate

Experiment - Padding and stride

Problem:You are training a convolutional neural network (CNN) on image data. The current model uses a convolution layer with stride=1 and 'valid' padding. The training accuracy is 95%, but the validation accuracy is only 70%. This suggests the model might be losing important edge information due to no padding, causing overfitting on the center features.

Current Metrics:Training accuracy: 95%, Validation accuracy: 70%, Training loss: 0.15, Validation loss: 0.60

Issue:The model is overfitting and validation accuracy is low because the convolution reduces image size too much, losing edge information. No padding and stride=1 cause this.

Your Task

Improve validation accuracy to above 80% by adjusting padding and stride to better preserve image features and reduce overfitting.

Keep the same number of convolutional layers and filters.

Only change padding and stride parameters in convolution layers.

Do not change the dataset or overall model architecture.

Hint 1

Hint 2

Hint 3

Solution

TensorFlow

import tensorflow as tf
from tensorflow.keras import layers, models

# Load example dataset
mnist = tf.keras.datasets.mnist
(X_train, y_train), (X_test, y_test) = mnist.load_data()

# Normalize and expand dims for channel
X_train = X_train.astype('float32') / 255.0
X_test = X_test.astype('float32') / 255.0
X_train = X_train[..., tf.newaxis]
X_test = X_test[..., tf.newaxis]

# Build model with 'same' padding and stride=2
model = models.Sequential([
    layers.Conv2D(32, kernel_size=3, strides=2, padding='same', activation='relu', input_shape=(28,28,1)),
    layers.MaxPooling2D(pool_size=2),
    layers.Flatten(),
    layers.Dense(64, activation='relu'),
    layers.Dense(10, activation='softmax')
])

model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

history = model.fit(X_train, y_train, epochs=5, batch_size=64, validation_split=0.2)

# Evaluate on test data
test_loss, test_acc = model.evaluate(X_test, y_test)

Changed padding from 'valid' to 'same' to keep feature map size and preserve edge information.

Changed stride from 1 to 2 to reduce feature map size and help reduce overfitting.

Kept other model parameters and layers unchanged.

Results Interpretation

Before: Training accuracy 95%, Validation accuracy 70%, Validation loss 0.60

After: Training accuracy 90%, Validation accuracy 82%, Validation loss 0.40

Using 'same' padding preserves image edges during convolution, preventing loss of important features. Increasing stride reduces feature map size, which can help reduce overfitting by simplifying the model's representation. Together, these changes improve validation accuracy and reduce overfitting.

Bonus Experiment

Try using 'same' padding with stride=1 but add dropout layers to reduce overfitting. Compare validation accuracy.

💡 Hint

Dropout randomly turns off neurons during training, which helps the model generalize better.

Practice

1. What does padding do in a convolutional layer in TensorFlow?

easy

A. Changes the filter size

B. Adjusts the learning rate

C. Controls the number of filters

D. Adds extra space around the input edges

Solution

1. Step 1: Understand padding role

   Padding adds pixels around the input image edges to control output size and preserve edge information.

2. Step 2: Compare options

   Only Adds extra space around the input edges correctly describes padding as adding space around input edges.

3. Final Answer:

   Adds extra space around the input edges -> Option D

4. Quick Check:

   Padding = Adds space around input edges [OK]

Hint: Padding adds border space around input images [OK]

Common Mistakes:

• Confusing padding with stride
• Thinking padding changes filter size
• Assuming padding controls learning rate

2. Which of the following is the correct way to specify stride of 2 in TensorFlow's Conv2D layer?

easy

A. strides=(2)

B. stride=(2)

C. strides=(2, 2)

D. stride=2

Solution

1. Step 1: Recall Conv2D stride syntax

   TensorFlow Conv2D expects strides as a tuple for height and width, e.g., (2, 2).

2. Step 2: Check options

   Only strides=(2, 2) uses the correct parameter name and tuple format.

3. Final Answer:

   strides=(2, 2) -> Option C

4. Quick Check:

   Stride syntax = strides=(2, 2) [OK]

Hint: Use tuple for strides in Conv2D: strides=(height, width) [OK]

Common Mistakes:

• Using 'stride' instead of 'strides'
• Passing a single integer instead of a tuple
• Omitting parentheses around stride values

3. Given an input of size 28x28, a filter size of 3x3, stride of 2, and 'valid' padding, what is the output size after Conv2D?

medium

A. 13x13

B. 14x14

C. 15x15

D. 12x12

Solution

1. Step 1: Calculate output size formula for 'valid' padding

   Output size = floor((Input - Filter) / Stride) + 1 = floor((28 - 3) / 2) + 1 = floor(25 / 2) + 1 = 12 + 1 = 13.

2. Step 2: Confirm output dimensions

   Output height and width are both 13, so output size is 13x13.

3. Final Answer:

   13x13 -> Option A

4. Quick Check:

   Output size = 13x13 with valid padding [OK]

Hint: Use formula: floor((input-filter)/stride)+1 for 'valid' padding [OK]

Common Mistakes:

• Using 'same' padding formula instead
• Forgetting to floor the division
• Adding padding when 'valid' means no padding

4. You wrote this code but get an error:

tf.keras.layers.Conv2D(filters=32, kernel_size=3, strides=2, padding='samee')

What is the error?

medium

A. Strides must be a tuple, not integer

B. Invalid padding value 'samee'

C. Kernel size must be a tuple

D. Filters must be a list

Solution

1. Step 1: Check padding parameter

   Padding accepts only 'valid' or 'same'. 'samee' is a typo and invalid.

2. Step 2: Verify other parameters

   Strides can be integer or tuple, kernel_size can be int or tuple, filters is integer. So no error there.

3. Final Answer:

   Invalid padding value 'samee' -> Option B

4. Quick Check:

   Padding typo causes error [OK]

Hint: Check spelling of padding values: 'same' or 'valid' only [OK]

Common Mistakes:

• Misspelling padding values
• Assuming strides must be tuple always
• Confusing kernel_size type requirements

5. You want to keep the output size same as input (28x28) after Conv2D with filter size 5x5. Which padding and stride should you use?

hard

A. padding='same', strides=(1, 1)

B. padding='valid', strides=(1, 1)

C. padding='same', strides=(2, 2)

D. padding='valid', strides=(2, 2)

Solution

1. Step 1: Understand output size preservation

   To keep output size same as input, use 'same' padding and stride 1 so filter slides over every pixel with padding added.

2. Step 2: Check options

   Only padding='same', strides=(1, 1) uses 'same' padding and stride (1,1), which preserves size.

3. Final Answer:

   padding='same', strides=(1, 1) -> Option A

4. Quick Check:

   Same padding + stride 1 keeps size [OK]

Hint: Use 'same' padding and stride 1 to keep output size [OK]

Common Mistakes:

• Using 'valid' padding which shrinks output
• Using stride > 1 which reduces output size
• Confusing filter size effect on output