Warmup strategies in PyTorch - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to create a linear warmup scheduler for the optimizer.

PyTorch

from torch.optim.lr_scheduler import LambdaLR
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
warmup_scheduler = LambdaLR(optimizer, lr_lambda=lambda step: step / [1] if step < 100 else 1)

Drag options to blanks, or click blank then click option'

A50

B10

C200

D100

Attempts:

3 left

💡 Hint

Common Mistakes

Using a denominator different from the warmup steps length.

Forgetting to set the learning rate to 1 after warmup.

The warmup scheduler increases the learning rate linearly until step 100, so the denominator must be 100.

2fill in blank

medium

Complete the code to apply a cosine warmup schedule with 50 warmup steps.

PyTorch

def cosine_warmup(step):
    if step < [1]:
        return step / 50
    else:
        return 0.5 * (1 + math.cos(math.pi * (step - 50) / (total_steps - 50)))

Drag options to blanks, or click blank then click option'

A25

B50

C100

D10

Attempts:

3 left

💡 Hint

Common Mistakes

Using the wrong number of warmup steps in the condition.

Not matching the denominator with the warmup steps.

The warmup phase is 50 steps, so the condition checks if step < 50.

3fill in blank

hard

Fix the error in the warmup scheduler code to correctly update the learning rate.

PyTorch

for epoch in range(epochs):
    for step, batch in enumerate(dataloader):
        optimizer.zero_grad()
        loss = model(batch)
        loss.backward()
        optimizer.step()
        scheduler.[1]()

Drag options to blanks, or click blank then click option'

Astep()

Bstep

Cstep_step

Dstep_step()

Attempts:

3 left

💡 Hint

Common Mistakes

Calling scheduler.step without parentheses.

Using an incorrect method name.

The scheduler's step method must be called as a function with parentheses: scheduler.step()

4fill in blank

hard

Fill both blanks to create a warmup scheduler that multiplies the learning rate by a factor during warmup and then decays it.

PyTorch

def warmup_decay(step):
    if step < [1]:
        return [2] * step
    else:
        return 0.1

Drag options to blanks, or click blank then click option'

A0.01

B100

C50

D0.1

Attempts:

3 left

💡 Hint

Common Mistakes

Confusing the warmup steps with the decay factor.

Using the decay factor in place of the warmup multiplier.

The warmup lasts 50 steps, and the learning rate is multiplied by 0.01 times the step during warmup.

5fill in blank

hard

Fill all three blanks to create a dictionary comprehension that maps each epoch to its learning rate using warmup and decay.

PyTorch

lr_schedule = {epoch: ([1] * epoch if epoch < [2] else [3]) for epoch in range(1, 101)}

Drag options to blanks, or click blank then click option'

A0.02

B50

C0.001

D100

Attempts:

3 left

💡 Hint

Common Mistakes

Mixing up the warmup length and decay learning rate.

Using incorrect multipliers for learning rate.

During warmup (first 50 epochs), learning rate increases by 0.02 * epoch; after that, it decays to 0.001.