Learning rate differential in PyTorch - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to set the learning rate for the optimizer.

PyTorch

optimizer = torch.optim.SGD(model.parameters(), lr=[1])

Drag options to blanks, or click blank then click option'

A1.0

B0.1

C0.01

D10

Attempts:

3 left

💡 Hint

Common Mistakes

Using a learning rate that is too large, causing unstable training.

Using an integer instead of a decimal for learning rate.

The learning rate is set to 0.01, which is a common small value to start training.

2fill in blank

medium

Complete the code to create two parameter groups with different learning rates.

PyTorch

optimizer = torch.optim.Adam([
    {'params': model.base.parameters(), 'lr': [1],
    {'params': model.head.parameters(), 'lr': 0.01}
])

Drag options to blanks, or click blank then click option'

A0.05

B0.1

D0.001

Attempts:

3 left

💡 Hint

Common Mistakes

Setting both learning rates the same defeats the purpose of differential learning rates.

Using too large learning rate for base parameters.

The base model uses a smaller learning rate 0.001, while the head uses 0.01 for faster learning.

3fill in blank

hard

Fix the error in the learning rate scheduler step call.

PyTorch

scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=10, gamma=0.1)
for epoch in range(20):
    train()
    validate()
    [1]

Drag options to blanks, or click blank then click option'

Ascheduler.step()

Bscheduler.update()

Cscheduler.step(epoch)

Dscheduler.step(10)

Attempts:

3 left

💡 Hint

Common Mistakes

Passing an argument to step() causes errors.

Using a non-existent method like update().

The scheduler.step() method is called without arguments to update the learning rate after each epoch.

4fill in blank

hard

Fill both blanks to create a dictionary of learning rates for different layers.

PyTorch

lr_dict = {
    'base': [1],
    'head': [2]
}

Drag options to blanks, or click blank then click option'

A0.001

B0.01

C0.1

D0.0001

Attempts:

3 left

💡 Hint

Common Mistakes

Using the same learning rate for both layers.

Choosing too large learning rates for base layers.

The base layer uses 0.001 and the head uses 0.01 as learning rates for differential training.

5fill in blank

hard

Fill all three blanks to define optimizer with differential learning rates and weight decay.

PyTorch

optimizer = torch.optim.Adam([
    {'params': model.backbone.parameters(), 'lr': [1], 'weight_decay': [2],
    {'params': model.classifier.parameters(), 'lr': [3], 'weight_decay': 0.01}
])

Drag options to blanks, or click blank then click option'

A0.0005

B0.001

C0.0001

D0.0003

Attempts:

3 left

💡 Hint

Common Mistakes

Mixing up learning rates and weight decay values.

Using too large learning rates causing unstable training.

The backbone uses learning rate 0.0001 with weight decay 0.001, classifier uses 0.0003 learning rate and 0.01 weight decay.