CosineAnnealingLR in PyTorch - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is the purpose of the CosineAnnealingLR scheduler in PyTorch?

CosineAnnealingLR adjusts the learning rate following a cosine curve, gradually decreasing it to a minimum value to help the model converge better during training.

beginner

How does the learning rate change over time with CosineAnnealingLR?

The learning rate starts at the initial value and decreases following a half cosine wave until it reaches the minimum learning rate at the end of the cycle.

intermediate

What are the key parameters of CosineAnnealingLR in PyTorch?

The key parameters are 'optimizer' (the optimizer to adjust), 'T_max' (the number of iterations for one cycle), and 'eta_min' (the minimum learning rate).

intermediate

Why might you choose CosineAnnealingLR over a constant learning rate?

Because it helps the model avoid getting stuck in bad local minima by reducing the learning rate smoothly, which can improve training stability and final accuracy.

beginner

Show a simple PyTorch code snippet to create a CosineAnnealingLR scheduler.

import torch.optim as optim

optimizer = optim.SGD(model.parameters(), lr=0.1)
scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=10, eta_min=0.001)

for epoch in range(10):
    train()
    scheduler.step()

What does the 'T_max' parameter in CosineAnnealingLR represent?

A The number of iterations for one cosine cycle

B The maximum learning rate

C The minimum learning rate

D The optimizer type

'T_max' sets how many iterations or epochs the scheduler takes to complete one cosine annealing cycle.

What happens to the learning rate at the end of the CosineAnnealingLR cycle?

A It increases exponentially

B It becomes zero

C It resets to the initial learning rate

D It reaches the minimum learning rate 'eta_min'

At the end of the cycle, the learning rate reaches the minimum value specified by 'eta_min'.

Which optimizer can CosineAnnealingLR be used with?

A Any PyTorch optimizer

B Only Adam

C Only RMSprop

D Only SGD

CosineAnnealingLR works with any PyTorch optimizer passed to it.

Why is cosine annealing beneficial for training neural networks?

A It keeps the learning rate constant

B It smoothly decreases the learning rate to avoid sharp drops

C It increases the learning rate over time

D It randomly changes the learning rate

Cosine annealing smoothly decreases the learning rate, helping training stability and convergence.

What is the default value of 'eta_min' in CosineAnnealingLR if not specified?

A 0.1

B 0.001

C 0.0

D 1.0

By default, 'eta_min' is 0.0, meaning the learning rate can anneal down to zero.

Explain how the CosineAnnealingLR scheduler adjusts the learning rate during training.

Think about how the learning rate changes smoothly over time.

Describe a simple PyTorch training loop using CosineAnnealingLR.

Focus on where the scheduler fits in the training process.