StepLR and MultiStepLR in PyTorch - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of StepLR scheduler after 5 epochs

Given the following PyTorch learning rate scheduler code, what is the learning rate after 5 epochs?

PyTorch

import torch
optimizer = torch.optim.SGD([torch.nn.Parameter(torch.randn(2, 2, requires_grad=True))], lr=0.1)
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=3, gamma=0.1)
for epoch in range(5):
    scheduler.step()
current_lr = optimizer.param_groups[0]['lr']
print(current_lr)

A 0.01

B 0.1

C 0.001

D 0.0001

💡 Hint

StepLR reduces the learning rate by gamma every step_size epochs.

StepLR reduces the learning rate by multiplying it by gamma every step_size epochs. After 3 epochs, lr = 0.1 * 0.1 = 0.01. After 5 epochs, only one step happened, so lr remains 0.01.

❓ Predict Output

intermediate

Learning rate after 7 epochs with MultiStepLR

What is the learning rate after 7 epochs using this MultiStepLR scheduler?

PyTorch

import torch
optimizer = torch.optim.SGD([torch.nn.Parameter(torch.randn(2, 2, requires_grad=True))], lr=0.2)
scheduler = torch.optim.lr_scheduler.MultiStepLR(optimizer, milestones=[3, 6], gamma=0.5)
for epoch in range(7):
    scheduler.step()
current_lr = optimizer.param_groups[0]['lr']
print(current_lr)

A 0.2

B 0.05

C 0.1

D 0.025

💡 Hint

MultiStepLR multiplies lr by gamma at each milestone epoch.

At epoch 3, lr = 0.2 * 0.5 = 0.1. At epoch 6, lr = 0.1 * 0.5 = 0.05. After 7 epochs, lr remains 0.05.

❓ Model Choice

advanced

Choosing scheduler for gradual learning rate decay

You want a learning rate scheduler that reduces the learning rate smoothly every 2 epochs by a factor of 0.9. Which scheduler and parameters should you choose?

A StepLR with step_size=2 and gamma=0.9

B MultiStepLR with milestones=[2,4,6] and gamma=0.9

C StepLR with step_size=1 and gamma=0.9

D MultiStepLR with milestones=[1,3,5] and gamma=0.9

💡 Hint

StepLR reduces learning rate at fixed intervals.

StepLR with step_size=2 reduces the learning rate every 2 epochs by gamma=0.9, which matches the requirement for smooth decay every 2 epochs.

❓ Hyperparameter

advanced

Effect of gamma in MultiStepLR scheduler

If you set gamma=2 in a MultiStepLR scheduler, what happens to the learning rate at each milestone?

A The learning rate becomes zero

B The learning rate halves at each milestone

C The learning rate doubles at each milestone

D The learning rate stays the same

💡 Hint

Gamma multiplies the learning rate at milestones.

Gamma is a multiplier applied to the learning rate at each milestone. If gamma=2, the learning rate doubles at each milestone.

🔧 Debug

expert

Why does StepLR not reduce learning rate as expected?

A user writes this code but notices the learning rate does not change after epochs: import torch optimizer = torch.optim.Adam([torch.nn.Parameter(torch.randn(1, requires_grad=True))], lr=0.05) scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=3, gamma=0.1) for epoch in range(5): # training code here scheduler.step() print(f'Epoch {epoch+1}, lr: {optimizer.param_groups[0]["lr"]}') What is the likely cause?

A StepLR requires step_size to be 1 to work properly

B The initial learning rate is too low to observe changes

C scheduler.step() should be called before optimizer.step() in each epoch

D scheduler.step() should be called after optimizer.step() in each epoch

💡 Hint

Order of calling scheduler.step() affects learning rate update timing.

In PyTorch, scheduler.step() should be called after optimizer.step() to update the learning rate correctly for the next epoch. Calling scheduler.step() before optimizer.step() causes the learning rate to update too early, so the printed lr does not reflect the expected decay.

Practice

1. What is the main difference between StepLR and MultiStepLR in PyTorch?

easy

A. StepLR decreases learning rate at fixed intervals; MultiStepLR decreases at specific epochs.

B. StepLR increases learning rate; MultiStepLR decreases learning rate.

C. StepLR changes learning rate randomly; MultiStepLR keeps it constant.

D. StepLR is used only for batch size adjustment; MultiStepLR for learning rate.

Solution

1. Step 1: Understand StepLR behavior

   StepLR reduces the learning rate by a factor every fixed number of epochs (step size).

2. Step 2: Understand MultiStepLR behavior

   MultiStepLR reduces the learning rate at specific epochs defined by a list of milestones.

3. Final Answer:

   StepLR decreases learning rate at fixed intervals; MultiStepLR decreases at specific epochs. -> Option A

4. Quick Check:

   StepLR fixed steps, MultiStepLR specific milestones [OK]

Hint: StepLR uses fixed steps; MultiStepLR uses milestone epochs [OK]

Common Mistakes:

• Confusing increase vs decrease of learning rate
• Thinking StepLR changes learning rate randomly
• Mixing learning rate with batch size adjustments

2. Which of the following is the correct way to create a StepLR scheduler in PyTorch that reduces learning rate every 5 epochs by a factor of 0.1?

easy

A. scheduler = StepLR(optimizer, step_size=5, gamma=0.1)

B. scheduler = StepLR(optimizer, milestones=[5], gamma=0.1)

C. scheduler = MultiStepLR(optimizer, step_size=5, gamma=0.1)

D. scheduler = MultiStepLR(optimizer, milestones=[5], gamma=0.1)

Solution

1. Step 1: Recall StepLR parameters

   StepLR takes step_size (int) and gamma (decay factor).

2. Step 2: Identify correct syntax

   scheduler = StepLR(optimizer, step_size=5, gamma=0.1) uses step_size=5 and gamma=0.1, which matches the requirement.

3. Final Answer:

   scheduler = StepLR(optimizer, step_size=5, gamma=0.1) -> Option A

4. Quick Check:

   StepLR uses step_size, not milestones [OK]

Hint: StepLR uses step_size, MultiStepLR uses milestones list [OK]

Common Mistakes:

• Using milestones parameter with StepLR
• Confusing MultiStepLR and StepLR syntax
• Passing step_size as a list

3. Given the following code, what will be the learning rate after epoch 7?

optimizer = torch.optim.SGD(model.parameters(), lr=0.1)
scheduler = MultiStepLR(optimizer, milestones=[3, 6], gamma=0.1)
for epoch in range(8):
    scheduler.step()
    print(f"Epoch {epoch}: lr = {optimizer.param_groups[0]['lr']}")

medium

A. 0.01

B. 0.001

C. 0.1

D. 0.0001

Solution

1. Step 1: Understand milestones and gamma

   Learning rate reduces by factor 0.1 at epochs 3 and 6.

2. Step 2: Calculate learning rate at epoch 7

   Initial lr=0.1; after epoch 3: 0.1*0.1=0.01; after epoch 6: 0.01*0.1=0.001; so at epoch 7 lr=0.001.

3. Final Answer:

   0.001 -> Option B

4. Quick Check:

   Two milestones reduce lr twice: 0.1 -> 0.01 -> 0.001 [OK]

Hint: Multiply lr by gamma at each milestone passed [OK]

Common Mistakes:

• Forgetting to apply gamma at both milestones
• Assuming lr changes before first milestone
• Confusing StepLR with MultiStepLR behavior

4. Identify the error in this code snippet using StepLR:

optimizer = torch.optim.Adam(model.parameters(), lr=0.01)
scheduler = StepLR(optimizer, milestones=[10, 20], gamma=0.5)
for epoch in range(25):
    scheduler.step()
    print(optimizer.param_groups[0]['lr'])

medium

A. scheduler.step() must be called after optimizer.step() inside loop.

B. Optimizer Adam cannot be used with StepLR scheduler.

C. StepLR does not accept milestones parameter; use step_size instead.

D. Gamma value must be greater than 1 for StepLR.

Solution

1. Step 1: Check StepLR parameters

   StepLR expects step_size, not milestones.

2. Step 2: Identify misuse of milestones

   Passing milestones causes error; correct is step_size=10 for example.

3. Final Answer:

   StepLR does not accept milestones parameter; use step_size instead. -> Option C

4. Quick Check:

   StepLR uses step_size, not milestones [OK]

Hint: StepLR uses step_size, not milestones list [OK]

Common Mistakes:

• Using milestones with StepLR
• Thinking Adam optimizer is incompatible
• Misunderstanding gamma parameter range

5. You want to train a model for 30 epochs. You want the learning rate to drop by 0.1 at epochs 10 and 20, and then again every 5 epochs after epoch 20. Which scheduler setup correctly achieves this?

hard

A. Use StepLR with step_size=10 and gamma=0.1

B. Use StepLR with step_size=5 and gamma=0.1

C. Use MultiStepLR with milestones=[10, 20, 25, 30] and gamma=0.1

D. Use MultiStepLR with milestones=[10, 20] and gamma=0.1, then StepLR with step_size=5 after epoch 20

Solution

1. Step 1: Understand the requirement

   Learning rate drops at epochs 10 and 20, then every 5 epochs after 20 (i.e., 25, 30).

2. Step 2: Analyze scheduler options

   MultiStepLR can handle fixed milestones (10, 20). StepLR can handle regular steps (every 5 epochs). Combining both after epoch 20 fits the requirement.

3. Step 3: Evaluate options

   Use MultiStepLR with milestones=[10, 20, 25, 30] and gamma=0.1 misses epochs after 20 beyond 25 and 30; Use StepLR with step_size=5 and gamma=0.1 drops every 5 epochs from start; Use StepLR with step_size=10 and gamma=0.1 drops every 10 epochs only; Use MultiStepLR with milestones=[10, 20] and gamma=0.1, then StepLR with step_size=5 after epoch 20 correctly combines both schedulers.

4. Final Answer:

   Use MultiStepLR with milestones=[10, 20] and gamma=0.1, then StepLR with step_size=5 after epoch 20 -> Option D

5. Quick Check:

   Combine MultiStepLR for early milestones + StepLR for regular steps after [OK]

Hint: Combine MultiStepLR for milestones + StepLR for regular steps [OK]

Common Mistakes:

• Trying to use only one scheduler for mixed schedule
• Misplacing milestones or step_size values
• Assuming StepLR can handle irregular milestones