Kernel size, stride, padding in PyTorch - Deep Dive

Overview - Kernel size, stride, padding

What is it?

Kernel size, stride, and padding are key settings in convolutional neural networks that control how filters scan over input data like images. Kernel size is the size of the filter window that looks at parts of the input. Stride is how many steps the filter moves each time it slides. Padding adds extra space around the input edges to control output size and edge effects.

Why it matters

These settings decide how much detail the network sees and how big the output is after convolution. Without understanding them, models might lose important information or produce outputs too small to learn from. They help balance detail and computation, making deep learning practical and effective for tasks like image recognition.

Where it fits

Before learning this, you should know basic neural networks and what convolution means. After this, you can learn about pooling layers, dilation, and advanced convolution types like depthwise or transposed convolutions.

Mental Model

Core Idea

Kernel size, stride, and padding control how a filter moves over input data, shaping what the network sees and how big the output is.

Think of it like...

Imagine stamping a pattern on a large sheet of paper: kernel size is the stamp size, stride is how far you move the stamp each time, and padding is adding extra blank space around the paper so the stamp can reach edges.

Input (5x5) with padding=1 → padded input (7x7)
Kernel size=3x3
Stride=2

Sliding windows:
[■ ■ ■] → move 2 steps right → [■ ■ ■]
↓                         ↓
move 2 steps down → [■ ■ ■] → ...

Output size calculated by:
Output = floor((Input + 2*Padding - Kernel) / Stride) + 1

Build-Up - 7 Steps

1

Foundation

Understanding Kernel Size Basics

Concept: Kernel size defines the filter's height and width that scans the input.

In convolution, a kernel (or filter) is a small matrix that moves over the input image or feature map. The kernel size is usually square, like 3x3 or 5x5, meaning the filter looks at a 3 by 3 or 5 by 5 patch at a time. This size controls how much local information the filter captures.

Result

A 3x3 kernel looks at small local patches, capturing fine details; a larger kernel sees bigger patterns but with less detail.

Knowing kernel size helps you control the scale of features your model learns, from edges to textures.

2

Foundation

What Stride Means in Convolution

Concept: Stride controls how far the kernel moves after each step when scanning the input.

Stride is the number of pixels the kernel jumps over when sliding across the input. A stride of 1 means the kernel moves one pixel at a time, scanning every possible position. A stride of 2 skips every other pixel, making the output smaller and computation faster.

Result

Higher stride reduces output size and speeds up computation but may skip details.

Stride balances detail and efficiency by controlling how densely the kernel samples the input.

3

Intermediate

Role of Padding in Convolution

Concept: Padding adds extra pixels around the input edges to control output size and edge effects.

Without padding, the kernel cannot fully cover the edges of the input, causing the output to shrink. Padding adds zeros (or other values) around the input border so the kernel can slide over edges. Common types are 'valid' (no padding) and 'same' (padding to keep output size equal to input).

Result

Padding preserves spatial size or controls how much the output shrinks after convolution.

Padding prevents losing information at edges and helps maintain consistent output sizes.

4

Intermediate

Calculating Output Size from Parameters

🤔Before reading on: Do you think increasing padding always increases output size? Commit to yes or no.

Concept: Output size depends on input size, kernel size, stride, and padding using a formula.

The output height and width are calculated as: Output = floor((Input + 2 * Padding - Kernel) / Stride) + 1 This formula helps predict how big the output feature map will be after convolution.

Result

You can plan network layers to get desired output sizes and avoid errors.

Understanding this formula lets you design networks that fit your data and computational limits.

5

Intermediate

Effect of Kernel Size and Stride Together

🤔Before reading on: Does increasing stride always reduce output size more than increasing kernel size? Commit to yes or no.

Concept: Kernel size and stride together control the resolution and size of the output feature map.

A larger kernel covers more input area per step, while a larger stride skips more positions. Both reduce output size but affect feature extraction differently. Large kernels with small stride capture broad patterns densely; small kernels with large stride sample fewer positions but keep detail local.

Result

Choosing kernel and stride together balances detail and computational cost.

Knowing their combined effect helps tune models for accuracy and speed.

6

Advanced

Padding Types and Their Impact

🤔Before reading on: Is zero padding the only way to pad inputs? Commit to yes or no.

Concept: Different padding methods affect how edges are treated and can influence model performance.

Common padding methods include zero padding (adding zeros), reflection padding (mirroring edge pixels), and replication padding (repeating edge pixels). Zero padding is simplest but can create artificial edges. Reflection or replication padding can reduce edge artifacts and improve learning.

Result

Choosing padding type can improve model accuracy on edge features.

Understanding padding types helps avoid edge distortions that confuse the model.

7

Expert

Surprising Effects of Stride and Padding in Practice

🤔Before reading on: Can stride and padding choices cause the output to have unexpected sizes or lose important features? Commit to yes or no.

Concept: Stride and padding interact in complex ways that can cause subtle bugs or performance drops if not carefully chosen.

In practice, using stride > 1 with padding can cause the output to be smaller than expected or misaligned with input features. This can lead to loss of spatial information or difficulty in upsampling later. Also, asymmetric padding (different padding on each side) can shift features. Careful calculation and testing are needed.

Result

Models may behave unexpectedly if stride and padding are not balanced, causing training issues or poor accuracy.

Knowing these subtle interactions prevents common production bugs and helps design robust architectures.

Under the Hood

Convolution slides the kernel matrix over the input data, multiplying overlapping values and summing them to produce one output value per position. Stride controls the step size of this sliding. Padding extends the input borders with extra values (usually zeros) so the kernel can cover edge pixels fully. Internally, this process is a series of dot products between kernel weights and input patches, repeated across spatial dimensions.

Why designed this way?

These parameters were designed to control the receptive field size and output dimensions flexibly. Early CNNs used fixed kernel sizes and no padding, but this caused rapid shrinking of feature maps. Padding was introduced to preserve spatial dimensions, and stride was added to reduce computation and control output resolution. Alternatives like dilated convolutions exist but kernel size, stride, and padding remain fundamental for their simplicity and effectiveness.

Input (H x W)
  │
  ├─[Padding: add zeros around edges]
  │
  ├─[Kernel: small filter slides over input]
  │    ├─Moves by Stride steps
  │    └─At each position, multiply and sum
  │
  └─Output (calculated size)

Flow:
┌───────────────┐
│   Input Data  │
└──────┬────────┘
       │ Padding
       ▼
┌───────────────┐
│ Padded Input  │
└──────┬────────┘
       │ Slide Kernel by Stride
       ▼
┌───────────────┐
│ Convolution   │
│ (Multiply &   │
│  Sum)        │
└──────┬────────┘
       │ Output
       ▼
┌───────────────┐
│ Feature Map   │
└───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does padding always increase the output size? Commit to yes or no.

Common Belief: Padding always makes the output bigger than the input.

Reality: Padding can preserve output size or reduce shrinkage but does not increase output beyond input size plus padding.

Why it matters: Assuming padding increases output can lead to wrong layer size calculations and model errors.

Quick: Does a larger kernel size always mean better feature extraction? Commit to yes or no.

Common Belief: Bigger kernels always capture better features because they see more input at once.

Reality: Larger kernels capture broader patterns but may miss fine details and increase computation, sometimes hurting performance.

Why it matters: Blindly increasing kernel size wastes resources and can reduce model accuracy.

Quick: Does stride only affect output size, not feature quality? Commit to yes or no.

Common Belief: Stride just shrinks output size without affecting what features the model learns.

Reality: Stride changes sampling density, which can skip important details and affect feature quality.

Why it matters: Ignoring stride's effect on features can cause models to miss critical information.

Quick: Is zero padding the only padding method used in practice? Commit to yes or no.

Common Belief: Zero padding is the standard and only practical padding method.

Reality: Reflection and replication padding are also used to reduce edge artifacts and improve learning.

Why it matters: Using only zero padding can cause edge distortions that hurt model accuracy.

Expert Zone

1

Padding can be asymmetric, adding different amounts on each side, which shifts feature alignment subtly.

2

Stride greater than one can cause aliasing effects, losing spatial resolution and causing artifacts.

3

Kernel size interacts with dilation rate, changing the effective receptive field without increasing parameters.

When NOT to use

Avoid large strides or no padding when precise spatial localization is needed, such as in segmentation. Instead, use dilated convolutions or transposed convolutions for upsampling. For edge-sensitive tasks, consider reflection padding over zero padding.

Production Patterns

In production, models often use small kernels (3x3) with stride 1 and padding 'same' to preserve size, stacking many layers for depth. Stride 2 is used for downsampling instead of pooling. Padding types are chosen based on dataset characteristics to reduce edge artifacts.

Connections

Pooling Layers

Builds-on

Understanding kernel size and stride helps grasp how pooling reduces spatial size and extracts dominant features.

Signal Processing Filters

Same pattern

Convolution kernels in CNNs are like filters in signal processing that extract frequency or pattern information from signals.

Human Visual Attention

Analogy in function

Kernel scanning with stride and padding mimics how human eyes focus on parts of a scene, moving attention stepwise and filling in edges.

Common Pitfalls

#1 Output size shrinks unexpectedly causing dimension mismatch errors.

Wrong approach:

conv = nn.Conv2d(in_channels=3, out_channels=16, kernel_size=5, stride=2, padding=0)
# Input size 32x32
# Output size calculated incorrectly

Correct approach:

conv = nn.Conv2d(in_channels=3, out_channels=16, kernel_size=5, stride=2, padding=2)
# Padding added to preserve output size better

Root cause:

Not accounting for padding effect on output size leads to unexpected shrinkage.

#2 Using large stride with no padding causes loss of edge information.

Wrong approach:

conv = nn.Conv2d(3, 16, kernel_size=3, stride=3, padding=0)

Correct approach:

conv = nn.Conv2d(3, 16, kernel_size=3, stride=3, padding=1)

Root cause:

Ignoring padding when stride skips pixels causes edges to be ignored.

#3 Assuming zero padding is always best for all tasks.

Wrong approach:

padding_mode='zeros' in all conv layers without testing alternatives

Correct approach:

padding_mode='reflect' or 'replicate' used for edge-sensitive tasks

Root cause:

Lack of awareness of padding types and their impact on edge artifacts.

Key Takeaways

Kernel size controls the area of input each filter looks at, affecting feature scale.

Stride determines how far the filter moves each step, balancing detail and speed.

Padding adds borders to inputs to preserve output size and protect edge information.

Output size depends on input size, kernel size, stride, and padding via a simple formula.

Choosing these parameters carefully is crucial to build effective and efficient convolutional networks.