Freezing layers in PyTorch - Deep Dive

Overview - Freezing layers

What is it?

Freezing layers means stopping some parts of a neural network from learning during training. When layers are frozen, their values do not change, so the model keeps what it already knows in those parts. This is useful when you want to keep some knowledge fixed and only train other parts. It helps save time and avoid forgetting useful information.

Why it matters

Freezing layers lets us reuse knowledge from a model trained on one task to help with a new task. Without freezing, training might erase what the model already learned, making it slower or less accurate. This is important in real life when data is limited or training is expensive. It helps build smarter AI faster and with less data.

Where it fits

Before learning freezing layers, you should understand how neural networks train and update weights using gradients. After this, you can learn transfer learning, fine-tuning, and how to build efficient models by combining frozen and trainable parts.

Mental Model

Core Idea

Freezing layers means locking some parts of a neural network so they don’t change during training, preserving their learned knowledge.

Think of it like...

Imagine a cookbook where some recipes are perfect and you don’t want to change them, so you put a clear plastic cover over those pages. You can still write new recipes on uncovered pages, but the covered ones stay exactly the same.

┌───────────────┐
│ Neural Network│
│ ┌───────────┐ │
│ │ Layer 1   │ │  ← Frozen (locked, no change)
│ ├───────────┤ │
│ │ Layer 2   │ │  ← Frozen (locked, no change)
│ ├───────────┤ │
│ │ Layer 3   │ │  ← Trainable (can update)
│ └───────────┘ │
└───────────────┘

Build-Up - 7 Steps

1

Foundation

What are neural network layers

Concept: Layers are building blocks of neural networks that transform input data step-by-step.

A neural network is made of layers. Each layer has weights that change during training to learn patterns. For example, an image passes through layers that detect edges, shapes, and objects. Training means adjusting these weights to improve predictions.

Result

Understanding layers helps you see where learning happens in a model.

Knowing layers are the parts that learn is key to understanding how freezing affects training.

2

Foundation

How training updates layers

Concept: Training changes layer weights using gradients to reduce errors.

When training, the model guesses outputs and compares them to true answers. It calculates errors and uses gradients to adjust weights in each layer to improve. This process repeats many times, making the model better.

Result

Weights in all layers change unless told otherwise.

Recognizing that training changes weights everywhere sets the stage for why freezing is needed.

3

Intermediate

What freezing layers means in practice

🤔Before reading on: do you think freezing layers means removing them or just stopping their weights from changing? Commit to your answer.

Concept: Freezing means stopping weight updates in some layers while keeping them in the model.

In PyTorch, freezing a layer means setting its weights to not require gradients. This tells the training process to skip updating those weights. The layer still processes data but stays fixed.

Result

Frozen layers keep their learned knowledge unchanged during training.

Understanding freezing as stopping updates, not removing layers, clarifies how models keep old knowledge.

4

Intermediate

How to freeze layers in PyTorch

🤔Before reading on: do you think freezing layers requires changing the model architecture or just a simple flag? Commit to your answer.

Concept: Freezing layers is done by setting requires_grad=False for their parameters.

Example code: import torch import torchvision.models as models model = models.resnet18(pretrained=True) # Freeze all layers for param in model.parameters(): param.requires_grad = False # Unfreeze last layer for param in model.fc.parameters(): param.requires_grad = True This freezes all layers except the last fully connected layer.

Result

Only the last layer's weights will update during training.

Knowing freezing is a simple flag change helps quickly control training behavior.

5

Intermediate

Why freeze layers during transfer learning

🤔Before reading on: do you think freezing layers helps or hurts learning new tasks? Commit to your answer.

Concept: Freezing preserves useful features learned on old tasks while adapting only parts needed for new tasks.

When using a pretrained model on a new task, freezing early layers keeps general features like edges intact. Training only later layers adapts the model to the new task without losing old knowledge. This saves time and data.

Result

Models learn new tasks faster and more reliably.

Understanding freezing as knowledge preservation explains why transfer learning works well.

6

Advanced

Partial freezing and fine-tuning strategies

🤔Before reading on: do you think freezing is all-or-nothing or can be done selectively? Commit to your answer.

Concept: Freezing can be applied to some layers while others remain trainable, enabling fine-tuning.

You can freeze early layers and train middle or last layers. For example, freeze layers 1-5, train layers 6-10. This balances keeping old knowledge and learning new details. Fine-tuning often starts with freezing most layers, then gradually unfreezing more.

Result

Fine-tuning improves performance on new tasks without overfitting.

Knowing freezing is flexible allows better control over model adaptation.

7

Expert

Surprising effects of freezing on training dynamics

🤔Before reading on: do you think freezing layers always speeds up training? Commit to your answer.

Concept: Freezing layers changes gradient flow and can affect training speed and stability in unexpected ways.

Freezing layers reduces the number of parameters updated, which can speed up training. However, it can also cause gradients to vanish or explode in unfrozen layers if not managed well. Sometimes, freezing too many layers leads to poor convergence or suboptimal solutions. Careful layer selection and learning rate tuning are needed.

Result

Freezing can both help and hinder training depending on setup.

Understanding freezing’s impact on gradients prevents common training pitfalls and improves model tuning.

Under the Hood

When a layer’s parameters have requires_grad=False, PyTorch’s autograd engine skips computing gradients for them during backpropagation. This means no gradient updates happen for those weights, so their values stay fixed. The forward pass still uses these weights to compute outputs. This selective gradient blocking lets parts of the model remain static while others learn.

Why designed this way?

Freezing was designed to enable transfer learning and efficient training by reusing pretrained knowledge. Instead of retraining entire large models, freezing allows focusing compute on new parts. Early deep learning research showed that lower layers learn general features useful across tasks, so freezing them saves time and data. Alternatives like copying weights or pruning were less flexible or efficient.

┌───────────────┐
│ Forward Pass  │
│  ┌─────────┐  │
│  │ Layer 1 │  │  ← Uses fixed weights
│  ├─────────┤  │
│  │ Layer 2 │  │  ← Uses fixed weights
│  ├─────────┤  │
│  │ Layer 3 │  │  ← Uses trainable weights
│  └─────────┘  │
└─────┬─────────┘
      │
┌─────▼─────────┐
│ Backpropagation│
│  ┌─────────┐  │
│  │ Layer 1 │  │  ← No gradients computed
│  ├─────────┤  │
│  │ Layer 2 │  │  ← No gradients computed
│  ├─────────┤  │
│  │ Layer 3 │  │  ← Gradients computed
│  └─────────┘  │
└───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does freezing layers mean removing them from the model? Commit to yes or no.

Common Belief: Freezing layers means deleting or skipping those layers during training.

Reality: Frozen layers still exist and process data; only their weights do not update.

Why it matters: Removing layers changes model behavior and can break predictions, while freezing preserves learned features.

Quick: Does freezing layers always make training faster? Commit to yes or no.

Common Belief: Freezing layers always speeds up training because fewer weights update.

Reality: Freezing can speed up training but may also cause slower convergence or instability if gradients behave poorly.

Why it matters: Assuming freezing always helps can lead to poor training choices and wasted time.

Quick: Can you freeze layers after training starts? Commit to yes or no.

Common Belief: Freezing layers must be done before training begins and cannot be changed later.

Reality: You can freeze or unfreeze layers anytime during training to adjust learning.

Why it matters: Knowing this allows flexible training strategies like gradual unfreezing for better results.

Quick: Does freezing layers mean the model forgets old knowledge? Commit to yes or no.

Common Belief: Frozen layers lose their learned knowledge because they don’t update.

Reality: Frozen layers keep their learned weights exactly, preserving old knowledge.

Why it matters: Misunderstanding this can cause unnecessary retraining or discarding useful pretrained models.

Expert Zone

1

Freezing layers affects optimizer state; some optimizers keep momentum which can cause unexpected updates if not reset.

2

Batch normalization layers behave differently when frozen; their running statistics may need special handling to avoid performance drops.

3

Gradual unfreezing, where layers are unfrozen one by one during training, often yields better fine-tuning results than freezing all then unfreezing suddenly.

When NOT to use

Freezing is not ideal when the new task is very different from the original or when full model adaptation is needed. In such cases, training all layers or using other techniques like distillation or reinitialization may be better.

Production Patterns

In production, freezing is used to speed up training on new data, reduce overfitting, and deploy models with fixed feature extractors. Common patterns include freezing backbone CNN layers in vision tasks and fine-tuning only classifier heads.

Connections

Transfer learning

Freezing layers is a core technique used in transfer learning to reuse pretrained knowledge.

Understanding freezing clarifies how transfer learning adapts models efficiently without retraining everything.

Gradient descent optimization

Freezing layers modifies which parameters receive gradient updates during optimization.

Knowing freezing’s effect on gradients helps understand training dynamics and optimizer behavior.

Software version control

Freezing layers is like locking files in version control to prevent changes while others evolve.

This cross-domain link shows how controlling change is a universal concept in managing complexity.

Common Pitfalls

#1 Forgetting to set requires_grad=False for frozen layers.

Wrong approach:

for param in model.parameters():
pass # No freezing done

# Training updates all weights

Correct approach:

for param in model.parameters():
param.requires_grad = False

# Frozen layers won’t update

Root cause:

Not knowing requires_grad controls gradient computation leads to ineffective freezing.

#2 Freezing batch normalization layers without adjusting their mode.

Wrong approach:

for param in model.bn.parameters():
param.requires_grad = False
model.train() # Keeps batch norm in training mode

Correct approach:

for param in model.bn.parameters():
param.requires_grad = False
model.eval() # Sets batch norm to evaluation mode

Root cause:

Ignoring batch norm’s running stats causes performance drops when frozen but left in training mode.

#3 Freezing all layers when the new task needs full adaptation.

Wrong approach:

for param in model.parameters():
param.requires_grad = False
# No layers trainable for new task

Correct approach:

for param in model.parameters():
param.requires_grad = True
# Train all layers for full adaptation

Root cause:

Misjudging task similarity leads to freezing too much and poor learning.

Key Takeaways

Freezing layers means stopping some parts of a neural network from updating during training to preserve learned knowledge.

In PyTorch, freezing is done by setting requires_grad=False on layer parameters, which prevents gradient updates.

Freezing is essential in transfer learning to reuse pretrained features and speed up training on new tasks.

Freezing can be applied selectively to balance preserving old knowledge and learning new information through fine-tuning.

Understanding freezing’s effects on gradients and training dynamics helps avoid common pitfalls and improve model performance.