Padding and stride in TensorFlow - ML Experiment: Train & Evaluate

Experiment - Padding and stride

Problem:You are training a convolutional neural network (CNN) on image data. The current model uses a convolution layer with stride=1 and 'valid' padding. The training accuracy is 95%, but the validation accuracy is only 70%. This suggests the model might be losing important edge information due to no padding, causing overfitting on the center features.

Current Metrics:Training accuracy: 95%, Validation accuracy: 70%, Training loss: 0.15, Validation loss: 0.60

Issue:The model is overfitting and validation accuracy is low because the convolution reduces image size too much, losing edge information. No padding and stride=1 cause this.

Your Task

Improve validation accuracy to above 80% by adjusting padding and stride to better preserve image features and reduce overfitting.

Keep the same number of convolutional layers and filters.

Only change padding and stride parameters in convolution layers.

Do not change the dataset or overall model architecture.

Hint 1

Hint 2

Hint 3

Solution

TensorFlow

import tensorflow as tf
from tensorflow.keras import layers, models

# Load example dataset
mnist = tf.keras.datasets.mnist
(X_train, y_train), (X_test, y_test) = mnist.load_data()

# Normalize and expand dims for channel
X_train = X_train.astype('float32') / 255.0
X_test = X_test.astype('float32') / 255.0
X_train = X_train[..., tf.newaxis]
X_test = X_test[..., tf.newaxis]

# Build model with 'same' padding and stride=2
model = models.Sequential([
    layers.Conv2D(32, kernel_size=3, strides=2, padding='same', activation='relu', input_shape=(28,28,1)),
    layers.MaxPooling2D(pool_size=2),
    layers.Flatten(),
    layers.Dense(64, activation='relu'),
    layers.Dense(10, activation='softmax')
])

model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

history = model.fit(X_train, y_train, epochs=5, batch_size=64, validation_split=0.2)

# Evaluate on test data
test_loss, test_acc = model.evaluate(X_test, y_test)

Changed padding from 'valid' to 'same' to keep feature map size and preserve edge information.

Changed stride from 1 to 2 to reduce feature map size and help reduce overfitting.

Kept other model parameters and layers unchanged.

Results Interpretation

Before: Training accuracy 95%, Validation accuracy 70%, Validation loss 0.60

After: Training accuracy 90%, Validation accuracy 82%, Validation loss 0.40

Using 'same' padding preserves image edges during convolution, preventing loss of important features. Increasing stride reduces feature map size, which can help reduce overfitting by simplifying the model's representation. Together, these changes improve validation accuracy and reduce overfitting.

Bonus Experiment

Try using 'same' padding with stride=1 but add dropout layers to reduce overfitting. Compare validation accuracy.

💡 Hint

Dropout randomly turns off neurons during training, which helps the model generalize better.