ResNet and skip connections in Computer Vision - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Why do ResNet models use skip connections?

ResNet introduced skip connections to help deep neural networks. What is the main reason for using these skip connections?

A To allow gradients to flow directly and avoid vanishing gradient problems in very deep networks.

B To reduce the number of parameters by skipping some layers during training.

C To increase the size of the input images automatically.

D To replace convolutional layers with fully connected layers.

💡 Hint

Think about what happens to gradients when networks get very deep.

Skip connections let gradients flow directly through the network, which helps avoid the vanishing gradient problem and allows training of very deep networks.

❓ Predict Output

intermediate

Output shape after a ResNet skip connection block

Consider a ResNet block where the input tensor has shape (batch_size=32, height=64, width=64, channels=64). The block applies two convolution layers with padding='same' and keeps the number of channels the same. What will be the output shape after adding the skip connection?

Computer Vision

import tensorflow as tf
input_tensor = tf.random.normal([32, 64, 64, 64])
conv1 = tf.keras.layers.Conv2D(64, 3, padding='same', activation='relu')(input_tensor)
conv2 = tf.keras.layers.Conv2D(64, 3, padding='same')(conv1)
output = tf.keras.layers.Add()([input_tensor, conv2])
print(output.shape)

A (32, 32, 32, 64)

B (32, 62, 62, 64)

C (32, 64, 64, 128)

D (32, 64, 64, 64)

💡 Hint

Padding='same' keeps height and width unchanged. The skip connection adds tensors of the same shape.

Since padding='same' keeps spatial dimensions and channels unchanged, adding the input tensor and conv2 output results in the same shape as input.

❓ Model Choice

advanced

Choosing the correct ResNet block for channel mismatch

In ResNet, when the input and output channels differ, a skip connection cannot be a simple addition. Which option correctly handles this channel mismatch?

A Add zero padding channels to the input tensor to match output channels.

B Use max pooling on the output to reduce channels before addition.

C Use a 1x1 convolution on the input to match output channels before addition.

D Skip the addition and concatenate input and output tensors instead.

💡 Hint

Think about how to change the input tensor shape to match the output tensor shape for addition.

A 1x1 convolution adjusts the input channels to match the output channels, enabling element-wise addition in the skip connection.

❓ Metrics

advanced

Effect of skip connections on training loss curves

When training a very deep ResNet with skip connections, how does the training loss curve typically compare to a similar deep network without skip connections?

A The ResNet with skip connections shows slower loss decrease and higher final training loss.

B The ResNet with skip connections usually shows faster loss decrease and lower final training loss.

C Both networks show identical loss curves because skip connections do not affect training.

D The network without skip connections converges faster due to simpler architecture.

💡 Hint

Consider how skip connections help gradients during backpropagation.

Skip connections help gradients flow better, allowing faster convergence and lower training loss in deep networks.

🔧 Debug

expert

Identifying the error in a ResNet skip connection implementation

Examine the following PyTorch code snippet for a ResNet block with a skip connection. What error will occur when running this code?

Computer Vision

import torch
import torch.nn as nn

class ResNetBlock(nn.Module):
    def __init__(self, in_channels, out_channels):
        super().__init__()
        self.conv1 = nn.Conv2d(in_channels, out_channels, kernel_size=3, padding=1)
        self.conv2 = nn.Conv2d(out_channels, out_channels, kernel_size=3, padding=1)
        self.relu = nn.ReLU()

    def forward(self, x):
        out = self.relu(self.conv1(x))
        out = self.conv2(out)
        out += x  # skip connection
        out = self.relu(out)
        return out

block = ResNetBlock(64, 128)
input_tensor = torch.randn(1, 64, 32, 32)
output = block(input_tensor)

A RuntimeError due to shape mismatch when adding tensors of different channel sizes.

B SyntaxError because of missing colon in class definition.

C TypeError because ReLU cannot be applied to convolution layers.

D No error; the code runs correctly and outputs tensor shape (1, 128, 32, 32).

💡 Hint

Check the shapes of tensors before the addition in the forward method.

The input tensor has 64 channels but the output of conv2 has 128 channels. Adding them directly causes a shape mismatch error.

Practice

1. What is the main purpose of skip connections in a ResNet model?

easy

A. To replace convolutional layers with fully connected layers

B. To reduce the number of layers in the network

C. To allow information to flow directly across layers, helping training

D. To increase the size of the input images

Solution

1. Step 1: Understand skip connections role

   Skip connections let the input bypass some layers and add directly to the output, helping information flow.

2. Step 2: Connect to training deep networks

   This helps avoid problems like vanishing gradients, making training deep networks easier and more accurate.

3. Final Answer:

   To allow information to flow directly across layers, helping training -> Option C

4. Quick Check:

   Skip connections improve training by direct flow [OK]

Hint: Skip connections let info skip layers to ease training [OK]

Common Mistakes:

• Thinking skip connections reduce layers
• Confusing skip connections with input size changes
• Assuming skip connections replace convolution

2. Which of the following is the correct way to add a skip connection in PyTorch between input tensor x and output tensor out?

easy

A. out = x - out

B. out = x * out

C. out = x / out

D. out = x + out

Solution

1. Step 1: Recall skip connection operation

   Skip connections add the input tensor to the output tensor element-wise.

2. Step 2: Match with correct syntax

   The addition operation out = x + out correctly implements the skip connection.

3. Final Answer:

   out = x + out -> Option D

4. Quick Check:

   Skip connection = addition [OK]

Hint: Skip connections use addition, not multiplication or division [OK]

Common Mistakes:

• Using multiplication instead of addition
• Using subtraction or division which breaks skip connection
• Confusing order of operands

3. Consider this PyTorch code snippet for a ResNet block:

import torch
import torch.nn as nn

class SimpleResBlock(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv = nn.Conv2d(3, 3, kernel_size=3, padding=1)
        self.relu = nn.ReLU()
        self.conv.weight.data.fill_(0.0)
        self.conv.bias.data.fill_(1.0)

    def forward(self, x):
        out = self.conv(x)
        out = self.relu(out)
        out = out + x
        return out

block = SimpleResBlock()
input_tensor = torch.ones(1, 3, 5, 5)
output = block(input_tensor)
print(output[0,0,0,0].item())

What will be printed?

medium

A. 2.0

B. 1.0

C. 0.0

D. An error occurs

Solution

1. Step 1: Analyze convolution output

   The convolution with kernel size 3 and padding 1 keeps the input size. Since input is all ones, convolution output will be some positive values (not zero).

2. Step 2: Add input and apply ReLU

   ReLU keeps positive values. Then adding input tensor (all ones) increases values. So output values > 1.

3. Final Answer:

   2.0 -> Option A

4. Quick Check:

   Output = conv + input > 1 [OK]

Hint: Skip connection adds input, so output > input value [OK]

Common Mistakes:

• Assuming output equals input without addition
• Ignoring padding effect on size
• Expecting zero or error due to shape mismatch

4. You wrote this PyTorch code for a ResNet block but get a runtime error:

def forward(self, x):
    out = self.conv(x)
    out = self.relu(out)
    out = out + x
    return out

The error says: "The size of tensor a (64) must match the size of tensor b (128) at non-singleton dimension 1." What is the likely cause?

medium

A. The convolution changes the number of channels, so shapes don't match for addition

B. ReLU changes tensor shape unexpectedly

C. Input tensor is None

D. The addition operator is used incorrectly

Solution

1. Step 1: Understand error message

   The error says channel sizes differ (64 vs 128), so tensors can't be added element-wise.

2. Step 2: Check convolution output channels

   If convolution changes channels from 64 to 128, input and output shapes differ, causing addition error.

3. Final Answer:

   The convolution changes the number of channels, so shapes don't match for addition -> Option A

4. Quick Check:

   Channel mismatch causes addition error [OK]

Hint: Check channel sizes before adding tensors [OK]

Common Mistakes:

• Blaming ReLU for shape errors
• Ignoring channel dimension mismatch
• Assuming addition works regardless of shape

5. In a ResNet architecture, if the input tensor has shape (batch_size, 64, 32, 32) and the convolution layer in the block changes channels to 128 with stride 2, how can you correctly implement the skip connection?

hard

A. Add input tensor directly without changes

B. Use a 1x1 convolution with stride 2 on the input to match shape before addition

C. Use max pooling on output tensor before addition

D. Skip connection is not needed in this case

Solution

1. Step 1: Identify shape mismatch

   Input has 64 channels and size 32x32; output has 128 channels and size 16x16 due to stride 2.

2. Step 2: Match shapes for addition

   To add tensors, input must be transformed to 128 channels and 16x16 size, done by 1x1 convolution with stride 2.

3. Final Answer:

   Use a 1x1 convolution with stride 2 on the input to match shape before addition -> Option B

4. Quick Check:

   Match shape with 1x1 conv before skip add [OK]

Hint: Use 1x1 conv to match shape for skip connection [OK]

Common Mistakes:

• Adding tensors with different shapes directly
• Using pooling on output instead of input
• Skipping skip connection when channels differ