Freezing layers in PyTorch - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the output of this PyTorch code snippet?

Consider the following PyTorch code that freezes some layers of a model. What will be the value of requires_grad for each parameter after running this code?

PyTorch

import torch
import torch.nn as nn

class SimpleModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(10, 5)
        self.fc2 = nn.Linear(5, 2)

    def forward(self, x):
        x = self.fc1(x)
        x = self.fc2(x)
        return x

model = SimpleModel()

# Freeze fc1 layer
for param in model.fc1.parameters():
    param.requires_grad = False

requires_grad_list = [param.requires_grad for param in model.parameters()]
print(requires_grad_list)

A [True, False, True, False]

B [True, True, False, False]

C [False, False, True, True]

D [False, True, False, True]

💡 Hint

Remember that each Linear layer has two parameters: weights and biases.

The fc1 layer has two parameters (weight and bias), both set to requires_grad=False. The fc2 layer's parameters remain trainable (requires_grad=True).

❓ Model Choice

intermediate

Which model freezing approach correctly freezes only the convolutional layers?

You want to freeze all convolutional layers in a PyTorch model but keep other layers trainable. Which code snippet correctly achieves this?

A

for name, param in model.named_parameters():
    if 'conv' in name:
        param.requires_grad = False

B

for layer in model.children():
    if 'conv' in str(layer):
        layer.requires_grad = False

C

for param in model.parameters():
    if isinstance(param, nn.Conv2d):
        param.requires_grad = False

D

for module in model.modules():
    if isinstance(module, nn.Conv2d):
        for param in module.parameters():
            param.requires_grad = False

💡 Hint

Parameters themselves are tensors, not layers. You need to access modules to check their type.

Option D correctly iterates over modules, checks if they are convolutional layers, then freezes their parameters. Option D only checks parameter names, which may be unreliable. Option D tries to check parameter type, which is invalid. Option D tries to set requires_grad on layers, which is not a valid attribute.

❓ Hyperparameter

advanced

What is the effect of freezing layers on optimizer hyperparameters?

You freeze some layers in your PyTorch model by setting requires_grad=False for their parameters. What should you do with the optimizer to avoid updating frozen parameters?

A Pass only parameters with <code>requires_grad=True</code> to the optimizer.

B Pass all model parameters to the optimizer; it will automatically skip frozen ones.

C Set the learning rate to zero for frozen parameters in the optimizer.

D Freeze layers after optimizer creation; no changes needed.

💡 Hint

Optimizer updates parameters it receives. Frozen parameters should not be included.

Only parameters with requires_grad=True should be passed to the optimizer. Passing frozen parameters wastes memory and may cause errors.

🔧 Debug

advanced

Why does training still update frozen layers?

You froze some layers by setting requires_grad=False, but after training, those layers' weights changed. What is the most likely cause?

A The loss function does not support frozen layers, causing updates.

B You forgot to filter parameters passed to the optimizer; optimizer still updates frozen layers.

C You used torch.no_grad() during training, which disables freezing.

D The model layers were not set to eval() mode, so gradients were computed anyway.

💡 Hint

Freezing layers disables gradient computation, but optimizer updates parameters it receives.

If frozen parameters are still passed to the optimizer, their values can be updated by optimizer steps even if gradients are zero or None.

🧠 Conceptual

expert

What is the impact of freezing layers on transfer learning performance?

When using transfer learning, freezing early layers of a pretrained model is common. What is the main reason for freezing these layers?

A Early layers capture general features; freezing them prevents overfitting and speeds up training.

B Early layers are specific to the original task; freezing them improves adaptation to new data.

C Freezing early layers increases model capacity by adding more trainable parameters.

D Freezing early layers reduces model size by removing those layers from computation.

💡 Hint

Think about what early layers learn in deep networks and why retraining them might be unnecessary.

Early layers learn general features like edges and textures useful across tasks. Freezing them reduces training time and risk of overfitting on small datasets.

Practice

1. What does freezing layers in a PyTorch model do during training?

easy

A. Removes the layers from the model

B. Increases the learning rate for those layers

C. Stops the layers' weights from updating

D. Duplicates the layers for faster training

Solution

1. Step 1: Understand freezing layers meaning

   Freezing layers means preventing their weights from changing during training.

2. Step 2: Effect on training

   When frozen, layers do not update weights, so they keep learned features intact.

3. Final Answer:

   Stops the layers' weights from updating -> Option C

4. Quick Check:

   Freezing = no weight updates [OK]

Hint: Freezing means no weight changes during training [OK]

Common Mistakes:

• Thinking freezing increases learning rate
• Believing freezing removes layers
• Assuming freezing duplicates layers

2. Which of the following is the correct way to freeze all parameters in a PyTorch model named model?

easy

A. model.freeze()

B. for param in model.parameters(): param.requires_grad = False

C. model.requires_grad = False

D. for param in model.parameters(): param.grad = None

Solution

1. Step 1: Identify correct syntax to freeze parameters

   Freezing requires setting requires_grad = False for each parameter.

2. Step 2: Check each option

   for param in model.parameters(): param.requires_grad = False correctly loops over parameters and sets requires_grad = False. Others are invalid or incorrect.

3. Final Answer:

   for param in model.parameters(): param.requires_grad = False -> Option B

4. Quick Check:

   Set requires_grad False to freeze [OK]

Hint: Set requires_grad=False on each parameter to freeze [OK]

Common Mistakes:

• Using model.requires_grad instead of param.requires_grad
• Calling a non-existent freeze() method
• Setting param.grad to None does not freeze

3. Consider this PyTorch code snippet:

import torch.nn as nn
model = nn.Sequential(
  nn.Linear(10, 5),
  nn.ReLU(),
  nn.Linear(5, 2)
)
for param in model[0].parameters():
  param.requires_grad = False

trainable_params = [p for p in model.parameters() if p.requires_grad]
print(len(trainable_params))

What will be printed?

medium

A. 2

B. 4

C. 6

D. 0

Solution

1. Step 1: Analyze model layers and parameters

   model[0] is Linear(10,5) with 2 parameters (weight and bias). model[2] is Linear(5,2) with 2 parameters.

2. Step 2: Check which parameters are trainable

   Parameters in model[0] are frozen (requires_grad=False), so only model[2]'s 2 parameters remain trainable.

3. Final Answer:

   2 -> Option A

4. Quick Check:

   Frozen layer params excluded, trainable = 2 [OK]

Hint: Count only parameters with requires_grad=True [OK]

Common Mistakes:

• Counting all parameters ignoring requires_grad
• Assuming ReLU has parameters
• Confusing layer indices

4. You want to freeze the first layer of a PyTorch model but accidentally wrote:

for param in model.layer1.parameters():
    param.grad = False

What is the problem with this code?

medium

A. param.grad is a tensor, not a boolean flag

B. param.grad disables gradients correctly

C. model.layer1.parameters() does not exist

D. param.requires_grad should be set, not param.grad

Solution

1. Step 1: Understand difference between param.grad and param.requires_grad

   param.grad holds gradient values, it is a tensor or None, not a flag to enable/disable gradients.

2. Step 2: Correct way to freeze parameters

   To freeze, set param.requires_grad = False. Setting param.grad = False is invalid and does not freeze.

3. Final Answer:

   param.requires_grad should be set, not param.grad -> Option D

4. Quick Check:

   Freeze by requires_grad=False, not param.grad [OK]

Hint: Freeze with requires_grad, not param.grad [OK]

Common Mistakes:

• Confusing param.grad with requires_grad
• Trying to disable gradients by setting param.grad
• Assuming param.grad is a boolean

5. You have a pretrained PyTorch model with 3 layers: layer1, layer2, and layer3. You want to freeze layer1 and layer2 but train layer3. Which code correctly freezes only the first two layers?

hard

A. for layer in [model.layer1, model.layer2]:
for param in layer.parameters():
param.requires_grad = False

B. for param in model.parameters():
param.requires_grad = False
for param in model.layer3.parameters():
param.requires_grad = False

C. model.layer1.requires_grad = False
model.layer2.requires_grad = False

D. model.freeze_layers(['layer1', 'layer2'])

Solution

1. Step 1: Understand freezing multiple layers

   Freezing means setting requires_grad = False on each parameter in the layers to freeze.

2. Step 2: Evaluate options

   for layer in [model.layer1, model.layer2]: for param in layer.parameters(): param.requires_grad = False correctly loops over layer1 and layer2 parameters and freezes them correctly. for param in model.parameters(): param.requires_grad = False for param in model.layer3.parameters(): param.requires_grad = False incorrectly freezes all parameters including layer3. model.layer1.requires_grad = False model.layer2.requires_grad = False tries to set requires_grad on layers (invalid). model.freeze_layers(['layer1', 'layer2']) calls a non-existent method.

3. Final Answer:

   for layer in [model.layer1, model.layer2]: for param in layer.parameters(): param.requires_grad = False -> Option A

4. Quick Check:

   Freeze layers by setting requires_grad False per parameter [OK]

Hint: Freeze layers by looping params and setting requires_grad False [OK]

Common Mistakes:

• Setting requires_grad on layer objects instead of parameters
• Using non-existent freeze_layers method
• Freezing all parameters