Kernel size, stride, padding in PyTorch - Model Metrics & Evaluation

Metrics & Evaluation - Kernel size, stride, padding

Which metric matters for Kernel size, stride, padding and WHY

When working with convolution layers, the key metrics to understand are the output size and receptive field. These depend on kernel size, stride, and padding.

Output size tells us how big the feature map will be after convolution. It affects model speed and memory.

Receptive field is how much of the input each output pixel 'sees'. Larger receptive fields capture more context.

We measure output size using this formula:
Output = floor((Input + 2 * Padding - Kernel Size) / Stride) + 1

Understanding these helps us design models that balance detail and efficiency.

Confusion matrix or equivalent visualization

For kernel size, stride, and padding, a confusion matrix is not used. Instead, we visualize how these parameters affect output size.

Input size: 7
Kernel size: 3
Stride: 2
Padding: 1

Output size = floor((7 + 2*1 - 3) / 2) + 1 = floor((7 + 2 - 3)/2) + 1 = floor(6/2) + 1 = 3 + 1 = 4

Visual:
Input: 7 pixels
Kernel slides over input with stride 2 and padding 1
Output: 4 pixels
    

Precision vs Recall tradeoff analogy for Kernel size, stride, padding

Think of kernel size, stride, and padding like taking photos of a scene:

• Kernel size is the size of your camera lens. Bigger lens sees more detail but takes longer to process.
• Stride is how far you move the camera between shots. Bigger stride means fewer shots, faster but less detail.
• Padding is adding extra space around the scene so you don't miss edges.

Tradeoff:

• Big kernel + small stride + padding = detailed but slow and large output.
• Small kernel + big stride + no padding = fast but may miss details at edges.

Choosing these is like balancing speed and detail in your photos.

What "good" vs "bad" values look like for Kernel size, stride, padding

Good values:

• Kernel size: Usually 3 or 5 for good detail without too much cost.
• Stride: 1 or 2 to keep enough output resolution.
• Padding: Enough to keep output size reasonable and preserve edges.

Bad values:

• Kernel size too large (e.g., 11) can over-smooth and slow down training.
• Stride too large (e.g., 4 or more) causes output to be too small, losing detail.
• No padding when needed causes output to shrink too much and lose edge info.

Common pitfalls with Kernel size, stride, padding

• Not calculating output size correctly leads to shape mismatch errors.
• Using no padding causes output to shrink after each layer, losing info.
• Too large stride skips important details, hurting accuracy.
• Ignoring receptive field size can cause model to miss important context.
• Changing these parameters without adjusting others can break model architecture.

Self-check question

Your convolution layer has input size 28, kernel size 5, stride 1, and no padding. What is the output size? Is this good if you want to keep input size?

Calculate output size:
Output = floor((28 + 2*0 - 5) / 1) + 1 = floor(23) + 1 = 24

Output size is 24, smaller than input 28. So output shrinks.

If you want to keep input size, you should add padding = (kernel size - 1) / 2 = 2.

With padding 2:
Output = floor((28 + 2*2 - 5)/1) + 1 = floor(27) + 1 = 28

Now output size matches input size, which is good for many models.

Key Result

Output size depends on kernel size, stride, and padding; correct calculation ensures proper model design.

Practice

1. What does the stride parameter control in a convolutional layer in PyTorch?

easy

A. How far the filter moves on the input each step

B. The size of the filter scanning the input

C. The number of filters used in the layer

D. The amount of zero padding added around the input

Solution

1. Step 1: Understand stride in convolution

   Stride defines the step size the filter moves when scanning the input image or feature map.

2. Step 2: Differentiate stride from other parameters

   Kernel size is the filter size, padding adds pixels around input, and number of filters controls output depth.

3. Final Answer:

   How far the filter moves on the input each step -> Option A

4. Quick Check:

   Stride = step size of filter movement [OK]

Hint: Stride = filter step size, not size or padding [OK]

Common Mistakes:

• Confusing stride with kernel size
• Thinking stride controls padding
• Mixing stride with number of filters

2. Which of the following is the correct way to create a 2D convolutional layer in PyTorch with kernel size 3, stride 2, and padding 1?

easy

A. nn.Conv2d(in_channels=1, out_channels=10, kernel_size=3, stride=2, padding=1)

B. nn.Conv2d(1, 10, kernel=3, stride=2, pad=1)

C. nn.Conv2d(1, 10, kernel_size=3, stride=1, padding=2)

D. nn.Conv2d(in_channels=1, out_channels=10, kernel_size=2, stride=2, padding=1)

Solution

1. Step 1: Check PyTorch Conv2d parameter names

   Correct parameters are in_channels, out_channels, kernel_size, stride, and padding.

2. Step 2: Match values to question

   Kernel size=3, stride=2, padding=1 matches only nn.Conv2d(in_channels=1, out_channels=10, kernel_size=3, stride=2, padding=1) exactly.

3. Final Answer:

   nn.Conv2d(in_channels=1, out_channels=10, kernel_size=3, stride=2, padding=1) -> Option A

4. Quick Check:

   Correct parameter names and values used [OK]

Hint: Use exact PyTorch parameter names: kernel_size, stride, padding [OK]

Common Mistakes:

• Using wrong parameter names like kernel or pad
• Mixing stride and padding values
• Wrong kernel size or stride values

3. Given an input tensor of size (1, 1, 7, 7), a Conv2d layer with kernel_size=3, stride=2, and padding=1 is applied. What is the output spatial size (height and width)?

medium

A. (1, 1, 3, 3)

B. (1, 1, 5, 5)

C. (1, 1, 2, 2)

D. (1, 1, 4, 4)

Solution

1. Step 1: Use output size formula for Conv2d

   Output size = floor((Input + 2*padding - kernel_size)/stride) + 1

2. Step 2: Calculate output height and width

   Input=7, padding=1, kernel=3, stride=2
   Output = floor((7 + 2*1 - 3)/2) + 1 = floor((7 + 2 - 3)/2) + 1 = floor(6/2) + 1 = 3 + 1 = 4

3. Final Answer:

   (1, 1, 4, 4) -> Option D

4. Quick Check:

   Output size formula applied correctly [OK]

Hint: Apply formula: floor((I+2P-K)/S)+1 for each dimension [OK]

Common Mistakes:

• Forgetting to add padding twice
• Using ceil instead of floor
• Mixing stride and kernel size in formula

4. You wrote this PyTorch code but get an error:

nn.Conv2d(1, 10, kernel_size=3, stride=2, padding=0)

on input size (1, 1, 1, 1). What is the likely cause?

medium

A. Kernel size must be even number

B. Padding is too small for the input size causing negative output dimension

C. Stride value must be 1 for kernel size 3

D. Input channels and output channels are swapped

Solution

1. Step 1: Check output size with given parameters

   Output size = floor((1 + 2*0 - 3)/2) + 1 = floor((-2)/2) + 1 = floor(-1) + 1 = -1 + 1 = 0 (invalid)

2. Step 2: Consider if padding causes error

   Padding=0 on small input (1x1) causes the calculated output size to be zero, which PyTorch raises as a runtime error due to insufficient padding for the kernel size.

3. Final Answer:

   Padding is too small for the input size causing negative output dimension -> Option B

4. Quick Check:

   Padding too small -> invalid output size [OK]

Hint: Check if padding is too small for input size [OK]

Common Mistakes:

• Assuming stride must be 1 for kernel 3
• Thinking kernel size must be even
• Swapping input and output channels

5. You want to keep the output size the same as the input size (7x7) after a Conv2d layer with kernel_size=5 and stride=1. What padding value should you use?

hard

A. 1

B. 0

C. 2

D. 3

Solution

1. Step 1: Use formula for output size with stride=1

   Output size = Input size if padding = (kernel_size - 1) / 2

2. Step 2: Calculate padding

   Padding = (5 - 1) / 2 = 4 / 2 = 2

3. Final Answer:

   2 -> Option C

4. Quick Check:

   Padding = (kernel_size - 1)/2 keeps size same [OK]

Hint: Padding = (kernel_size - 1) / 2 for same size with stride 1 [OK]

Common Mistakes:

• Using padding 0 or 1 incorrectly
• Forgetting stride must be 1 for same size
• Using padding larger than needed