ResNet and skip connections in Computer Vision - Deep Dive

Overview - ResNet and skip connections

What is it?

ResNet, short for Residual Network, is a type of deep learning model designed to make very deep neural networks easier to train. It uses skip connections, which are shortcuts that let information jump over some layers. These skip connections help the model learn better by avoiding problems that happen when networks get too deep, like losing important signals. ResNet has been very successful in tasks like image recognition.

Why it matters

Without ResNet and skip connections, very deep neural networks would struggle to learn because of issues like vanishing gradients, where signals get too weak as they pass through many layers. This would limit how powerful and accurate models can become. ResNet allows us to build much deeper networks that learn better and solve complex problems like recognizing objects in photos or videos, improving technologies like self-driving cars and medical imaging.

Where it fits

Before learning ResNet, you should understand basic neural networks and convolutional neural networks (CNNs). After ResNet, you can explore advanced architectures like DenseNet, EfficientNet, or transformers for vision tasks. ResNet is a key step in understanding how to build and train very deep models effectively.

Mental Model

Core Idea

Skip connections let information flow directly across layers, helping deep networks learn by preserving important signals and making training easier.

Think of it like...

Imagine a long hiking trail with many checkpoints. Normally, you have to pass through every checkpoint in order, which can be tiring and slow. Skip connections are like shortcuts that let you jump ahead, skipping some checkpoints so you don’t get too tired and can reach the destination faster and fresher.

Input Layer
   │
[Conv Layer 1]
   │
[Conv Layer 2]───┐
   │             │
   └─────────────>+ (Skip Connection)
                 │
             [Add Layer]
                 │
             [Activation]
                 │
               Output

Build-Up - 7 Steps

1

Foundation

Understanding Deep Neural Networks

Concept: Deep neural networks are models with many layers that learn complex patterns from data.

A neural network is like a chain of simple math operations. Each layer transforms the input a little bit. When you stack many layers, the network can learn very detailed features, like edges, shapes, and objects in images. But as you add more layers, training becomes harder.

Result

You get a model that can learn complex tasks but may be difficult to train if too deep.

Knowing how layers build on each other helps understand why very deep networks can struggle without special design.

2

Foundation

Problems with Very Deep Networks

Concept: Very deep networks face issues like vanishing gradients, making training slow or ineffective.

When training deep networks, the signals used to update the model’s knowledge get smaller as they move backward through layers. This is called vanishing gradients. It means early layers learn very slowly or not at all, limiting the network’s power.

Result

Training deep networks becomes inefficient or fails, causing poor performance.

Understanding this problem explains why simply adding layers doesn’t always improve models.

3

Intermediate

Introducing Skip Connections

🤔Before reading on: do you think skipping layers helps or harms learning in deep networks? Commit to your answer.

Concept: Skip connections let information bypass some layers, helping preserve signals during training.

A skip connection adds the input of a layer directly to the output of a deeper layer. This means the network can learn changes (residuals) instead of full transformations. It helps keep important information flowing and makes training more stable.

Result

Networks with skip connections train faster and can be much deeper without losing performance.

Knowing that networks learn residuals rather than full mappings changes how we think about deep learning.

4

Intermediate

Residual Blocks in ResNet

🤔Before reading on: do you think residual blocks add complexity or simplify training? Commit to your answer.

Concept: Residual blocks are building units in ResNet that use skip connections to learn residual functions.

Each residual block has two or more layers and a skip connection that adds the block’s input to its output. This addition helps the network focus on learning the difference from the input, making it easier to optimize.

Result

Residual blocks enable very deep networks to train effectively and improve accuracy.

Understanding residual blocks reveals why ResNet can be much deeper than previous models.

5

Intermediate

Training Deep ResNet Models

🤔Before reading on: do you think deeper ResNets always perform better or is there a limit? Commit to your answer.

Concept: ResNet allows training of very deep models by stabilizing gradients and improving feature reuse.

With skip connections, gradients flow more easily backward, preventing vanishing. Also, features learned early can be reused later, improving learning efficiency. However, extremely deep ResNets may still face diminishing returns.

Result

Deep ResNets achieve state-of-the-art results in image tasks but require careful tuning.

Knowing the limits of depth helps balance model size and performance.

6

Advanced

Variations and Extensions of ResNet

🤔Before reading on: do you think all skip connections are simple additions or can they be more complex? Commit to your answer.

Concept: ResNet inspired many variants that modify skip connections or block designs for better performance.

Some models use bottleneck blocks with 1x1 convolutions to reduce computation. Others add attention mechanisms or change how skip connections combine features. These tweaks improve speed, accuracy, or adapt ResNet to new tasks.

Result

Variants of ResNet are widely used in practice, showing the flexibility of skip connections.

Understanding these variations helps adapt ResNet ideas to different problems.

7

Expert

Why Skip Connections Work: Theoretical Insights

🤔Before reading on: do you think skip connections only help gradients or also affect the function space the network can represent? Commit to your answer.

Concept: Skip connections change the optimization landscape and function space, making training easier and more expressive.

Skip connections create paths where gradients do not vanish, improving optimization. They also let the network represent identity functions easily, preventing degradation when adding layers. This means deeper networks can at least perform as well as shallower ones, avoiding accuracy drop.

Result

Skip connections enable stable training and better generalization in very deep networks.

Knowing the theoretical reasons behind skip connections explains why they revolutionized deep learning.

Under the Hood

Skip connections add the input of a block directly to its output, creating a residual mapping. During backpropagation, this addition provides a direct gradient path, preventing gradients from shrinking too much. This helps early layers learn effectively even in very deep networks. The network learns the difference between input and output (residual), which is often easier than learning the full transformation.

Why designed this way?

ResNet was designed to solve the degradation problem where deeper networks performed worse than shallower ones. Traditional deep networks struggled with vanishing gradients and optimization difficulties. Skip connections were introduced as a simple yet powerful way to let networks learn residual functions, making training stable and enabling much deeper architectures.

Input x
  │
  ├─> [Layer 1] ─> [Layer 2] ─> F(x)
  │                      │
  └──────────────────────┤
                         +
                         │
                      Output = F(x) + x

Myth Busters - 4 Common Misconceptions

Quick: Do skip connections mean the network ignores some layers? Commit to yes or no.

Common Belief: Skip connections let the network skip or ignore some layers entirely.

Reality: Skip connections add the input to the output of layers; they do not bypass or disable layers but help preserve information.

Why it matters: Thinking layers are ignored can lead to misunderstanding how the network learns and misusing skip connections.

Quick: Do skip connections always improve any neural network? Commit to yes or no.

Common Belief: Adding skip connections always makes any neural network better.

Reality: Skip connections help mostly in very deep networks; in shallow networks, they may not provide benefits and can add unnecessary complexity.

Why it matters: Misapplying skip connections can waste resources and complicate models without improving performance.

Quick: Do skip connections only help with gradient flow? Commit to yes or no.

Common Belief: Skip connections only help by improving gradient flow during training.

Reality: Skip connections also change the function space, allowing identity mappings and better feature reuse, which improves generalization.

Why it matters: Limiting understanding to gradients misses why skip connections improve model expressiveness and stability.

Quick: Do you think ResNet’s skip connections are unique to image tasks? Commit to yes or no.

Common Belief: Skip connections are only useful for image recognition tasks.

Reality: Skip connections are used in many domains like speech, language, and reinforcement learning to improve deep model training.

Why it matters: Believing skip connections are domain-specific limits their application and innovation in other fields.

Expert Zone

1

Skip connections can be identity mappings or use projection (1x1 convolutions) to match dimensions, which affects model capacity and training.

2

The placement and type of activation functions around skip connections influence gradient flow and model expressiveness.

3

Very deep ResNets sometimes use stochastic depth, randomly dropping blocks during training to improve generalization.

When NOT to use

Skip connections are less useful in shallow networks or models where layer outputs are very different in size or meaning. Alternatives like DenseNet’s concatenation or transformer architectures may be better for certain tasks.

Production Patterns

In production, ResNet variants are often combined with batch normalization, dropout, and learning rate schedules. They serve as backbone models for object detection, segmentation, and video analysis pipelines.

Connections

Highway Networks

Builds-on

Highway Networks introduced gated skip connections, which inspired ResNet’s simpler additive skip connections, showing evolution in deep network design.

Gradient Descent Optimization

Supports

Skip connections improve gradient flow, directly impacting how gradient descent updates model weights effectively in deep networks.

Human Brain Neural Pathways

Analogy in biology

Like skip connections, the brain has shortcut pathways that allow signals to bypass certain neurons, enabling faster and more efficient processing.

Common Pitfalls

#1 Adding skip connections without matching dimensions causes errors.

Wrong approach:

output = layer_output + input # when layer_output and input have different shapes

Correct approach:

projected_input = conv1x1(input) # match dimensions
output = layer_output + projected_input

Root cause:

Skip connections require the input and output to have the same shape; ignoring this causes shape mismatch errors.

#2 Placing activation functions before addition breaks residual learning.

Wrong approach:

output = activation(layer_output) + input

Correct approach:

output = activation(layer_output + input)

Root cause:

Activation should be applied after adding skip connection to preserve the residual learning property.

#3 Using skip connections in very shallow networks unnecessarily complicates the model.

Wrong approach:

Adding skip connections in a 3-layer network without benefit.

Correct approach:

Use standard layers without skip connections for shallow networks.

Root cause:

Skip connections mainly help with deep networks; applying them in shallow networks adds complexity without gains.

Key Takeaways

ResNet uses skip connections to let information flow directly across layers, solving training problems in very deep networks.

Skip connections help networks learn residual functions, which are easier to optimize than full transformations.

This design prevents vanishing gradients and allows building much deeper models that perform better on complex tasks.

Understanding skip connections is key to grasping modern deep learning architectures and their success.

Applying skip connections correctly requires matching dimensions and proper placement of activation functions.

Practice

1. What is the main purpose of skip connections in a ResNet model?

easy

A. To replace convolutional layers with fully connected layers

B. To reduce the number of layers in the network

C. To allow information to flow directly across layers, helping training

D. To increase the size of the input images

Solution

1. Step 1: Understand skip connections role

   Skip connections let the input bypass some layers and add directly to the output, helping information flow.

2. Step 2: Connect to training deep networks

   This helps avoid problems like vanishing gradients, making training deep networks easier and more accurate.

3. Final Answer:

   To allow information to flow directly across layers, helping training -> Option C

4. Quick Check:

   Skip connections improve training by direct flow [OK]

Hint: Skip connections let info skip layers to ease training [OK]

Common Mistakes:

• Thinking skip connections reduce layers
• Confusing skip connections with input size changes
• Assuming skip connections replace convolution

2. Which of the following is the correct way to add a skip connection in PyTorch between input tensor x and output tensor out?

easy

A. out = x - out

B. out = x * out

C. out = x / out

D. out = x + out

Solution

1. Step 1: Recall skip connection operation

   Skip connections add the input tensor to the output tensor element-wise.

2. Step 2: Match with correct syntax

   The addition operation out = x + out correctly implements the skip connection.

3. Final Answer:

   out = x + out -> Option D

4. Quick Check:

   Skip connection = addition [OK]

Hint: Skip connections use addition, not multiplication or division [OK]

Common Mistakes:

• Using multiplication instead of addition
• Using subtraction or division which breaks skip connection
• Confusing order of operands

3. Consider this PyTorch code snippet for a ResNet block:

import torch
import torch.nn as nn

class SimpleResBlock(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv = nn.Conv2d(3, 3, kernel_size=3, padding=1)
        self.relu = nn.ReLU()
        self.conv.weight.data.fill_(0.0)
        self.conv.bias.data.fill_(1.0)

    def forward(self, x):
        out = self.conv(x)
        out = self.relu(out)
        out = out + x
        return out

block = SimpleResBlock()
input_tensor = torch.ones(1, 3, 5, 5)
output = block(input_tensor)
print(output[0,0,0,0].item())

What will be printed?

medium

A. 2.0

B. 1.0

C. 0.0

D. An error occurs

Solution

1. Step 1: Analyze convolution output

   The convolution with kernel size 3 and padding 1 keeps the input size. Since input is all ones, convolution output will be some positive values (not zero).

2. Step 2: Add input and apply ReLU

   ReLU keeps positive values. Then adding input tensor (all ones) increases values. So output values > 1.

3. Final Answer:

   2.0 -> Option A

4. Quick Check:

   Output = conv + input > 1 [OK]

Hint: Skip connection adds input, so output > input value [OK]

Common Mistakes:

• Assuming output equals input without addition
• Ignoring padding effect on size
• Expecting zero or error due to shape mismatch

4. You wrote this PyTorch code for a ResNet block but get a runtime error:

def forward(self, x):
    out = self.conv(x)
    out = self.relu(out)
    out = out + x
    return out

The error says: "The size of tensor a (64) must match the size of tensor b (128) at non-singleton dimension 1." What is the likely cause?

medium

A. The convolution changes the number of channels, so shapes don't match for addition

B. ReLU changes tensor shape unexpectedly

C. Input tensor is None

D. The addition operator is used incorrectly

Solution

1. Step 1: Understand error message

   The error says channel sizes differ (64 vs 128), so tensors can't be added element-wise.

2. Step 2: Check convolution output channels

   If convolution changes channels from 64 to 128, input and output shapes differ, causing addition error.

3. Final Answer:

   The convolution changes the number of channels, so shapes don't match for addition -> Option A

4. Quick Check:

   Channel mismatch causes addition error [OK]

Hint: Check channel sizes before adding tensors [OK]

Common Mistakes:

• Blaming ReLU for shape errors
• Ignoring channel dimension mismatch
• Assuming addition works regardless of shape

5. In a ResNet architecture, if the input tensor has shape (batch_size, 64, 32, 32) and the convolution layer in the block changes channels to 128 with stride 2, how can you correctly implement the skip connection?

hard

A. Add input tensor directly without changes

B. Use a 1x1 convolution with stride 2 on the input to match shape before addition

C. Use max pooling on output tensor before addition

D. Skip connection is not needed in this case

Solution

1. Step 1: Identify shape mismatch

   Input has 64 channels and size 32x32; output has 128 channels and size 16x16 due to stride 2.

2. Step 2: Match shapes for addition

   To add tensors, input must be transformed to 128 channels and 16x16 size, done by 1x1 convolution with stride 2.

3. Final Answer:

   Use a 1x1 convolution with stride 2 on the input to match shape before addition -> Option B

4. Quick Check:

   Match shape with 1x1 conv before skip add [OK]

Hint: Use 1x1 conv to match shape for skip connection [OK]

Common Mistakes:

• Adding tensors with different shapes directly
• Using pooling on output instead of input
• Skipping skip connection when channels differ