CosineAnnealingLR in PyTorch - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

What does CosineAnnealingLR scheduler do during training?

Imagine you are training a model and using the CosineAnnealingLR scheduler. What best describes how the learning rate changes over time?

A The learning rate oscillates between a maximum and minimum value following a cosine curve over a set period.

B The learning rate increases exponentially during training to speed up convergence.

C The learning rate stays constant until a certain epoch, then drops suddenly to a lower value.

D The learning rate decreases linearly from the initial value to zero over the total number of epochs.

💡 Hint

Think about how cosine functions behave between 0 and pi.

CosineAnnealingLR adjusts the learning rate by following a cosine curve, smoothly decreasing it from the initial value to a minimum and then optionally restarting. This helps the model avoid getting stuck and can improve training.

❓ Predict Output

intermediate

What is the learning rate after 5 epochs?

Given this PyTorch code snippet using CosineAnnealingLR, what is the learning rate at epoch 5?

PyTorch

import torch
import torch.optim as optim

model_params = [torch.nn.Parameter(torch.randn(2, 2, requires_grad=True))]
optimizer = optim.SGD(model_params, lr=0.1)
scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=10, eta_min=0.01)

lrs = []
for epoch in range(6):
    scheduler.step()
    lrs.append(optimizer.param_groups[0]['lr'])

print(lrs[-1])

A 0.1

B 0.01

C 0.075

D 0.055

💡 Hint

CosineAnnealingLR formula: lr = eta_min + (initial_lr - eta_min) * (1 + cos(pi * epoch / T_max)) / 2

At epoch 5, the learning rate is calculated as 0.01 + (0.1 - 0.01) * (1 + cos(pi * 5 / 10)) / 2 = 0.01 + 0.09 * (1 + cos(pi/2)) / 2 = 0.01 + 0.09 * (1 + 0) / 2 = 0.01 + 0.045 = 0.055.

❓ Model Choice

advanced

Which scenario benefits most from using CosineAnnealingLR?

You want to train a deep neural network that tends to get stuck in local minima. Which training setup is best suited for CosineAnnealingLR?

A Training a small linear regression model with a fixed learning rate.

B Training a deep convolutional network where you want the learning rate to restart periodically to escape local minima.

C Training a model with very few epochs where learning rate decay is not needed.

D Training a model with a learning rate that should increase steadily during training.

💡 Hint

CosineAnnealingLR can be combined with restarts to help escape local minima.

CosineAnnealingLR is especially useful for deep networks where the learning rate is gradually reduced and can be restarted to help the model jump out of local minima and explore better solutions.

❓ Hyperparameter

advanced

What effect does increasing T_max have in CosineAnnealingLR?

In the CosineAnnealingLR scheduler, what happens if you increase the T_max parameter while keeping other settings constant?

A The learning rate decreases faster and reaches eta_min sooner.

B The learning rate oscillates more frequently between max and min values.

C The learning rate decreases more slowly and takes longer to reach eta_min.

D The learning rate stays constant for longer before decreasing.

💡 Hint

T_max controls the period of the cosine cycle.

T_max sets the number of epochs for one full cosine cycle. Increasing it makes the learning rate decrease more slowly, extending the time before it reaches the minimum value.

🔧 Debug

expert

Why does the learning rate not change as expected?

You wrote this code to use CosineAnnealingLR but the learning rate stays constant at 0.1 for all epochs. What is the most likely cause?

PyTorch

import torch
import torch.optim as optim

model_params = [torch.nn.Parameter(torch.randn(2, 2, requires_grad=True))]
optimizer = optim.SGD(model_params, lr=0.1)
scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=10, eta_min=0.01)

for epoch in range(5):
    # training code here
    print(f"Epoch {epoch} lr: {optimizer.param_groups[0]['lr']}")
    # forgot to call scheduler.step()

A The scheduler.step() function was not called inside the training loop.

B The model parameters are not passed correctly to the optimizer.

C The eta_min parameter is set too low to affect the learning rate.

D The optimizer learning rate was set too high initially.

💡 Hint

Schedulers need to be updated each epoch to change the learning rate.

If scheduler.step() is not called, the learning rate will not update and will stay at the initial value throughout training.