How to Freeze Layers in PyTorch: Simple Guide

To freeze layers in PyTorch, set requires_grad = False for the parameters of those layers. This prevents their weights from updating during training, effectively freezing them.

Syntax

Freezing layers in PyTorch involves setting the requires_grad attribute of the layer's parameters to False. This tells PyTorch not to compute gradients for these parameters during backpropagation.

Typical syntax:

• for param in model.layer.parameters(): — loop over parameters of the layer
• param.requires_grad = False — freeze the parameter

python

for param in model.layer.parameters():
    param.requires_grad = False

Example

This example shows how to freeze the first convolutional layer of a simple CNN model so its weights do not change during training.

python

import torch
import torch.nn as nn

# Define a simple CNN
class SimpleCNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 10, kernel_size=3)
        self.conv2 = nn.Conv2d(10, 20, kernel_size=3)
        self.fc = nn.Linear(20*24*24, 2)

    def forward(self, x):
        x = torch.relu(self.conv1(x))
        x = torch.relu(self.conv2(x))
        x = x.view(x.size(0), -1)
        x = self.fc(x)
        return x

model = SimpleCNN()

# Freeze the first conv layer
for param in model.conv1.parameters():
    param.requires_grad = False

# Check requires_grad status
conv1_grads = [p.requires_grad for p in model.conv1.parameters()]
conv2_grads = [p.requires_grad for p in model.conv2.parameters()]

print('conv1 requires_grad:', conv1_grads)
print('conv2 requires_grad:', conv2_grads)

Output

conv1 requires_grad: [False, False]
conv2 requires_grad: [True, True]

Common Pitfalls

Common mistakes when freezing layers include:

• Not setting requires_grad = False before optimizer creation, so optimizer still updates frozen layers.
• Freezing layers but forgetting to exclude their parameters from the optimizer.
• Trying to freeze layers by setting model.eval(), which only affects dropout and batchnorm behavior, not gradients.

python

import torch
import torch.nn as nn

model = nn.Linear(10, 2)

# Wrong: freezing after optimizer creation
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)
for param in model.parameters():
    param.requires_grad = False

# The optimizer still updates parameters because it holds references

# Right way:
model = nn.Linear(10, 2)
for param in model.parameters():
    param.requires_grad = False
optimizer = torch.optim.SGD(filter(lambda p: p.requires_grad, model.parameters()), lr=0.1)

Quick Reference

Summary tips for freezing layers in PyTorch:

• Set param.requires_grad = False for all parameters you want to freeze.
• Create the optimizer after freezing layers, passing only parameters with requires_grad=True.
• Use filter(lambda p: p.requires_grad, model.parameters()) to select trainable parameters.
• Remember model.eval() does not freeze weights; it changes layer behavior during inference.

Key Takeaways

Set requires_grad = False on layer parameters to freeze them in PyTorch.

Freeze layers before creating the optimizer to avoid updating frozen weights.

Use optimizer with only parameters that require gradients to train correctly.

model.eval() does not freeze layers; it only changes inference behavior.

Check requires_grad flags to confirm which layers are frozen.