Why learning rate strategy affects convergence in PyTorch - Test Your Understanding

Practice - 5 Tasks

1fill in blank

easy

Complete the code to set the learning rate for the optimizer.

PyTorch

optimizer = torch.optim.SGD(model.parameters(), lr=[1])

Drag options to blanks, or click blank then click option'

Aloss

Bmodel

C0.01

Depoch

Attempts:

3 left

💡 Hint

Common Mistakes

Using a variable name instead of a number for learning rate.

Setting learning rate too high or zero.

The learning rate is a number that controls how big the steps are when the model learns. Here, 0.01 is a common starting learning rate.

2fill in blank

medium

Complete the code to apply a learning rate scheduler that reduces the learning rate every 10 epochs.

PyTorch

scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=[1], gamma=0.1)

Drag options to blanks, or click blank then click option'

A10

B0.1

C20

D5

Attempts:

3 left

💡 Hint

Common Mistakes

Confusing gamma with step_size.

Using a float instead of an integer for step_size.

The step_size parameter controls how often the learning rate is reduced. Here, every 10 epochs.

3fill in blank

hard

Fix the error in the training loop to update the learning rate scheduler correctly.

PyTorch

for epoch in range(num_epochs):
    train(model, optimizer)
    loss = validate(model)
    [1]  # update learning rate scheduler

Drag options to blanks, or click blank then click option'

Amodel.train()

Boptimizer.step()

Closs.backward()

Dscheduler.step()

Attempts:

3 left

💡 Hint

Common Mistakes

Calling optimizer.step() instead of scheduler.step().

Forgetting to call scheduler.step() at all.

The scheduler.step() call updates the learning rate according to the schedule after each epoch.

4fill in blank

hard

Fill both blanks to create a dictionary comprehension that maps epoch numbers to learning rates using scheduler.get_last_lr().

PyTorch

lr_dict = {epoch: [1] for epoch in range(num_epochs) if epoch [2] 5}

Drag options to blanks, or click blank then click option'

Ascheduler.get_last_lr()[0]

B>

C<

Depoch

Attempts:

3 left

💡 Hint

Common Mistakes

Using epoch instead of scheduler.get_last_lr()[0] for learning rate.

Using wrong comparison operator in the condition.

We get the current learning rate from scheduler.get_last_lr()[0]. The condition filters epochs greater than 5.

5fill in blank

hard

Fill all three blanks to create a training loop that updates the optimizer, scheduler, and prints the learning rate.

PyTorch

for epoch in range(num_epochs):
    optimizer.zero_grad()
    output = model(data)
    loss = criterion(output, target)
    loss.backward()
    optimizer.[1]()
    scheduler.[2]()
    print(f"Epoch {epoch}: lr = {scheduler.get_last_lr()[[3]]}")

Drag options to blanks, or click blank then click option'

Astep

C0

Dzero_grad

Attempts:

3 left

💡 Hint

Common Mistakes

Calling zero_grad() instead of step() on optimizer.

Using wrong index for get_last_lr().

optimizer.step() updates weights, scheduler.step() updates learning rate, and index 0 gets the current learning rate.

Practice

1. What is the main role of the learning rate in training a PyTorch model?

easy

A. It determines the type of activation function used.

B. It decides the number of layers in the model.

C. It sets the batch size for training.

D. It controls the size of the steps the model takes to learn.

Solution

1. Step 1: Understand learning rate function

   The learning rate controls how much the model changes its weights after seeing each batch of data.

2. Step 2: Identify the correct role

   Among the options, only controlling step size matches the learning rate's role.

3. Final Answer:

   It controls the size of the steps the model takes to learn. -> Option D

4. Quick Check:

   Learning rate = step size [OK]

Hint: Learning rate = step size in learning [OK]

Common Mistakes:

• Confusing learning rate with batch size
• Thinking learning rate sets model layers
• Mixing learning rate with activation functions

2. Which PyTorch code snippet correctly creates an optimizer with a learning rate of 0.01?

easy

A. optimizer = torch.optim.SGD(model.parameters(), learningRate=0.01)

B. optimizer = torch.optim.Adam(model.parameters(), learning_rate=0.01)

C. optimizer = torch.optim.SGD(model.parameters(), lr=0.01)

D. optimizer = torch.optim.Adam(model.parameters(), rate=0.01)

Solution

1. Step 1: Check PyTorch optimizer syntax

   The correct argument for learning rate is 'lr', not 'learning_rate' or 'learningRate' or 'rate'.

2. Step 2: Identify correct code

   optimizer = torch.optim.SGD(model.parameters(), lr=0.01) uses 'lr=0.01' correctly with SGD optimizer.

3. Final Answer:

   optimizer = torch.optim.SGD(model.parameters(), lr=0.01) -> Option C

4. Quick Check:

   Use 'lr' for learning rate in PyTorch optimizers [OK]

Hint: Use 'lr' keyword for learning rate in PyTorch [OK]

Common Mistakes:

• Using 'learning_rate' instead of 'lr'
• Wrong capitalization like 'learningRate'
• Using 'rate' instead of 'lr'

3. Consider this PyTorch training loop snippet with a fixed learning rate of 0.1:

optimizer = torch.optim.SGD(model.parameters(), lr=0.1)
for epoch in range(3):
    optimizer.zero_grad()
    output = model(input)
    loss = loss_fn(output, target)
    loss.backward()
    optimizer.step()
    print(f"Epoch {epoch+1} loss: {loss.item():.4f}")

What is the likely effect of using a high fixed learning rate like 0.1 on convergence?

medium

A. The model may overshoot minima and fail to converge.

B. The model will converge faster without any issues.

C. The model will ignore the learning rate and converge normally.

D. The loss will always be zero from the first epoch.

Solution

1. Step 1: Understand effect of high learning rate

   A high learning rate can cause the model to take too large steps, missing the best solution and causing unstable training.

2. Step 2: Analyze options

   Only The model may overshoot minima and fail to converge. correctly describes overshooting and failure to converge due to high learning rate.

3. Final Answer:

   The model may overshoot minima and fail to converge. -> Option A

4. Quick Check:

   High learning rate = overshoot minima [OK]

Hint: High learning rate risks overshooting minima [OK]

Common Mistakes:

• Assuming high learning rate always speeds convergence
• Thinking learning rate is ignored by optimizer
• Believing loss is zero immediately

4. You have this PyTorch code using a learning rate scheduler:

optimizer = torch.optim.SGD(model.parameters(), lr=0.1)
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=2, gamma=0.5)
for epoch in range(4):
    optimizer.zero_grad()
    output = model(input)
    loss = loss_fn(output, target)
    loss.backward()
    optimizer.step()
    scheduler.step()
    print(f"Epoch {epoch+1} lr: {scheduler.get_last_lr()[0]:.4f}")

The printed learning rates are: 0.0500, 0.0500, 0.0250, 0.0250. What is wrong?

medium

A. Calling scheduler.step() after optimizer.step() causes learning rate to update too early.

B. The scheduler should be called before optimizer.step() to update correctly.

C. The learning rate is not changing because gamma is too small.

D. The step_size should be 1 to update every epoch.

Solution

1. Step 1: Understand StepLR behavior

   StepLR updates learning rate every 'step_size' epochs by multiplying by 'gamma'. It should be called before optimizer.step() to update the learning rate correctly for the current epoch.

2. Step 2: Analyze learning rate printout

   Learning rate halves too early (at epoch 1 instead of 2), indicating scheduler.step() is called too late.

3. Final Answer:

   The scheduler should be called before optimizer.step() to update correctly. -> Option B

4. Quick Check:

   Scheduler step timing affects lr update [OK]

Hint: Scheduler.step() timing affects learning rate update [OK]

Common Mistakes:

• Assuming gamma controls if lr changes or not
• Thinking step_size must be 1 always
• Calling scheduler.step() after optimizer.step() causes early update

5. You want to train a model that first learns quickly and then fine-tunes slowly. Which learning rate strategy in PyTorch best fits this goal?

hard

A. Use a StepLR scheduler to reduce learning rate after fixed epochs.

B. Use a constant learning rate throughout training.

C. Use a learning rate that increases over time.

D. Use no learning rate scheduler and manually change lr each epoch.

Solution

1. Step 1: Understand training phases

   Starting with a higher learning rate helps fast learning; lowering it later helps fine-tuning.

2. Step 2: Match strategy to goal

   StepLR reduces learning rate after set epochs, matching the goal of fast then slow learning.

3. Final Answer:

   Use a StepLR scheduler to reduce learning rate after fixed epochs. -> Option A

4. Quick Check:

   StepLR = fast then slow learning [OK]

Hint: StepLR reduces learning rate after epochs for fine-tuning [OK]

Common Mistakes:

• Thinking constant lr adapts learning speed
• Believing increasing lr helps fine-tuning
• Ignoring built-in schedulers and changing lr manually