Learning rate differential in PyTorch - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Why use different learning rates for different layers?

In training deep neural networks, why might we assign different learning rates to different layers?

A Because different learning rates help the optimizer skip some layers during training.

B Because early layers often learn general features and require smaller learning rates, while later layers learn task-specific features and can use larger learning rates.

C Because using the same learning rate for all layers always causes the model to diverge.

D Because some layers have more parameters and need slower updates to avoid overfitting.

💡 Hint

Think about how early layers and later layers in a neural network behave differently during training.

Early layers capture general patterns and should change slowly to preserve learned features, so they use smaller learning rates. Later layers adapt to specific tasks and can learn faster with larger learning rates.

❓ Predict Output

intermediate

Output of learning rate differential setup in PyTorch

What will be the learning rate of the parameters in model.layer1 and model.layer2 after this code runs?

PyTorch

import torch
import torch.nn as nn

class SimpleModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.layer1 = nn.Linear(10, 5)
        self.layer2 = nn.Linear(5, 2)

model = SimpleModel()
optimizer = torch.optim.SGD([
    {'params': model.layer1.parameters(), 'lr': 0.001},
    {'params': model.layer2.parameters(), 'lr': 0.01}
], momentum=0.9)

lrs = [group['lr'] for group in optimizer.param_groups]
print(lrs)

A [0.01, 0.001]

B [0.001]

C [0.01]

D [0.001, 0.01]

💡 Hint

Look at how the optimizer parameter groups are defined with different learning rates.

The optimizer has two parameter groups: one for layer1 with learning rate 0.001, and one for layer2 with learning rate 0.01. The printed list shows these learning rates in order.

❓ Hyperparameter

advanced

Choosing learning rates for differential training

You want to fine-tune a pretrained model by freezing early layers and training only the last few layers. Which learning rate setup is best?

A Set zero learning rate for frozen layers and a small learning rate for trainable layers.

B Set a high learning rate for all layers to speed up training.

C Set a small learning rate for frozen layers and a high learning rate for trainable layers.

D Set the same moderate learning rate for all layers regardless of freezing.

💡 Hint

Frozen layers should not update during training.

Frozen layers should have zero learning rate to prevent updates. Trainable layers should have a small learning rate to fine-tune without large jumps.

❓ Metrics

advanced

Effect of learning rate differential on training loss

During training with differential learning rates, you notice the loss decreases quickly at first but then plateaus. What is a likely cause?

A The learning rate for later layers is too low, slowing learning.

B The learning rates are perfectly balanced; plateau is normal.

C The learning rate for early layers is too high, causing instability.

D The optimizer momentum is set to zero, causing slow convergence.

💡 Hint

Consider which layers learn task-specific features and how their learning rate affects training speed.

If later layers have too low learning rates, they learn slowly, causing loss to plateau early. Adjusting their learning rate can help continue loss reduction.

🔧 Debug

expert

Why does this differential learning rate code cause an error?

What error does this PyTorch code raise when trying to set different learning rates?

PyTorch

import torch
import torch.nn as nn

model = nn.Sequential(
    nn.Linear(10, 5),
    nn.ReLU(),
    nn.Linear(5, 2)
)

optimizer = torch.optim.Adam([
    {'params': model[0].parameters(), 'lr': 0.001},
    {'params': model[1].parameters(), 'lr': 0.01}
])

A No error, code runs successfully

B RuntimeError: optimizer parameter groups must have 'params' key

C AttributeError: 'ReLU' object has no attribute 'parameters'

D TypeError: optimizer got an unexpected keyword argument 'lr'

💡 Hint

Check which layers have parameters and which do not.

Activation layers like ReLU do not have parameters, so calling parameters() on them raises an AttributeError.