Learning rate schedulers in PyTorch - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

What is the main purpose of a learning rate scheduler?

In training neural networks, why do we use learning rate schedulers?

A To randomly reset model weights at certain epochs to prevent overfitting.

B To adjust the learning rate during training to improve convergence and avoid overshooting minima.

C To increase the batch size dynamically during training for faster computation.

D To change the activation function of the model layers during training.

💡 Hint

Think about how changing the step size affects learning progress.

Learning rate schedulers adjust the learning rate during training to help the model converge better and avoid jumping over minima.

❓ Predict Output

intermediate

Output of learning rate after 3 epochs with StepLR

Given the PyTorch code below, what is the learning rate printed after 3 epochs?

PyTorch

import torch
import torch.optim as optim

model_params = [torch.nn.Parameter(torch.randn(2, 2, requires_grad=True))]
optimizer = optim.SGD(model_params, lr=0.1)
scheduler = optim.lr_scheduler.StepLR(optimizer, step_size=2, gamma=0.5)

for epoch in range(3):
    optimizer.step()
    scheduler.step()
    lr = optimizer.param_groups[0]['lr']
print(f"Learning rate after 3 epochs: {lr}")

A Learning rate after 3 epochs: 0.05

B Learning rate after 3 epochs: 0.1

C Learning rate after 3 epochs: 0.025

D Learning rate after 3 epochs: 0.2

💡 Hint

StepLR reduces the learning rate by gamma every step_size epochs.

StepLR reduces lr by 0.5 every 2 epochs. After 2 epochs lr=0.05, after 3 epochs still 0.05.

❓ Model Choice

advanced

Choosing a scheduler for cyclic learning rate

You want the learning rate to cyclically increase and decrease during training to help escape local minima. Which PyTorch scheduler should you choose?

A torch.optim.lr_scheduler.CyclicLR

B torch.optim.lr_scheduler.StepLR

C torch.optim.lr_scheduler.ExponentialLR

D torch.optim.lr_scheduler.ReduceLROnPlateau

💡 Hint

Look for a scheduler that explicitly cycles the learning rate.

CyclicLR cycles the learning rate between bounds, increasing and decreasing it during training.

❓ Metrics

advanced

Effect of learning rate scheduler on training loss

During training, you apply a learning rate scheduler that reduces the learning rate when validation loss plateaus. What effect do you expect on the training loss curve?

A Training loss will remain constant regardless of scheduler.

B Training loss will increase sharply after scheduler reduces learning rate.

C Training loss will decrease more smoothly and possibly reach a lower minimum.

D Training loss will oscillate wildly without any pattern.

💡 Hint

Reducing learning rate on plateau helps fine-tune the model.

Lowering learning rate when progress stalls helps the model settle into a better minimum, smoothing loss decrease.

🔧 Debug

expert

Why does this CosineAnnealingLR scheduler not reduce learning rate as expected?

Consider this PyTorch code snippet:

import torch
import torch.optim as optim

params = [torch.nn.Parameter(torch.randn(1, requires_grad=True))]
optimizer = optim.SGD(params, lr=0.1)
scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=5)

for epoch in range(10):
    optimizer.step()
    scheduler.step()
    print(f"Epoch {epoch+1}: lr = {optimizer.param_groups[0]['lr']}")

The learning rate resets after 5 epochs but does not decrease smoothly over 10 epochs as intended. What is the cause?

A CosineAnnealingLR requires gamma parameter to reduce learning rate, which is missing.

B The learning rate is fixed and cannot be changed by CosineAnnealingLR.

C The optimizer.step() must be called after scheduler.step() for correct learning rate update.

D The scheduler restarts after T_max epochs; to have smooth decay over 10 epochs, T_max should be set to 10.

💡 Hint

Check the meaning of T_max parameter in CosineAnnealingLR.

T_max defines the number of epochs for one cosine cycle. Setting T_max=5 causes the learning rate to restart every 5 epochs, not smooth over 10.