Optimizers (SGD, Adam) in PyTorch - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to create an SGD optimizer for the model parameters with a learning rate of 0.01.

PyTorch

optimizer = torch.optim.[1](model.parameters(), lr=0.01)

Drag options to blanks, or click blank then click option'

ASGD

BAdam

CRMSprop

DAdagrad

Attempts:

3 left

💡 Hint

Common Mistakes

Using Adam instead of SGD when the question asks specifically for SGD.

Misspelling the optimizer name.

The SGD optimizer is created using torch.optim.SGD. It updates parameters using stochastic gradient descent.

2fill in blank

medium

Complete the code to create an Adam optimizer with a learning rate of 0.001.

PyTorch

optimizer = torch.optim.[1](model.parameters(), lr=0.001)

Drag options to blanks, or click blank then click option'

ASGD

BAdadelta

CAdam

DAdamW

Attempts:

3 left

💡 Hint

Common Mistakes

Choosing SGD instead of Adam when adaptive learning is needed.

Confusing Adam with AdamW which is a variant.

The Adam optimizer is created using torch.optim.Adam. It adapts learning rates for each parameter.

3fill in blank

hard

Fix the error in the code to correctly update the optimizer after computing gradients.

PyTorch

loss.backward()
optimizer.[1]()

Drag options to blanks, or click blank then click option'

Astep

Bzero_grad

Cbackward

Dupdate

Attempts:

3 left

💡 Hint

Common Mistakes

Calling 'zero_grad' instead of 'step' after backward pass.

Trying to call 'update' which is not a valid method.

After computing gradients with loss.backward(), call optimizer.step() to update model parameters.

4fill in blank

hard

Fill both blanks to reset gradients and then update parameters correctly.

PyTorch

optimizer.[1]()
loss.backward()
optimizer.[2]()

Drag options to blanks, or click blank then click option'

Azero_grad

Bstep

Cbackward

Dupdate

Attempts:

3 left

💡 Hint

Common Mistakes

Calling 'step' before 'zero_grad'.

Using 'update' which is not a valid optimizer method.

First, clear old gradients with optimizer.zero_grad(). Then compute gradients with loss.backward(). Finally, update parameters with optimizer.step().

5fill in blank

hard

Fill all three blanks to create an Adam optimizer with weight decay, reset gradients, and update parameters.

PyTorch

optimizer = torch.optim.Adam(model.parameters(), lr=[1], weight_decay=[2])
optimizer.[3]()
loss.backward()
optimizer.step()

Drag options to blanks, or click blank then click option'

A0.001

B0.0001

Czero_grad

Dstep

Attempts:

3 left

💡 Hint

Common Mistakes

Using zero_grad after backward pass instead of before.

Confusing weight_decay with learning rate.

The Adam optimizer is created with learning rate 0.001 and weight decay 0.0001. Before backward pass, call optimizer.zero_grad() to clear gradients.

Practice

1. Which optimizer in PyTorch adapts the learning rate for each parameter automatically?

easy

A. Adam

B. SGD

C. RMSprop

D. Adagrad

Solution

1. Step 1: Understand optimizer behavior

   Adam optimizer adjusts learning rates for each parameter based on past gradients.

2. Step 2: Compare with SGD

   SGD uses a fixed learning rate for all parameters without adaptation.

3. Final Answer:

   Adam -> Option A

4. Quick Check:

   Adaptive learning rate = Adam [OK]

Hint: Adam adapts learning rates per parameter automatically [OK]

Common Mistakes:

• Confusing SGD as adaptive optimizer
• Thinking RMSprop is the only adaptive optimizer
• Assuming Adagrad is default in PyTorch

2. Which of the following is the correct way to create an SGD optimizer in PyTorch with learning rate 0.01?

easy

A. optimizer = torch.optim.SGD(model, lr=0.01)

B. optimizer = torch.SGD(model.parameters(), learning_rate=0.01)

C. optimizer = torch.optim.Adam(model.parameters(), lr=0.01)

D. optimizer = torch.optim.SGD(model.parameters(), lr=0.01)

Solution

1. Step 1: Check PyTorch optimizer syntax

   Correct syntax uses torch.optim.SGD with model.parameters() and lr argument.

2. Step 2: Identify errors in other options

   The option with torch.SGD(model.parameters(), learning_rate=0.01) uses wrong module and argument name; the option with Adam uses wrong optimizer; the option passing model instead of parameters is incorrect.

3. Final Answer:

   optimizer = torch.optim.SGD(model.parameters(), lr=0.01) -> Option D

4. Quick Check:

   Correct syntax = optimizer = torch.optim.SGD(model.parameters(), lr=0.01) [OK]

Hint: Use torch.optim.SGD with model.parameters() and lr= value [OK]

Common Mistakes:

• Passing model instead of model.parameters()
• Using wrong argument name like learning_rate
• Confusing Adam with SGD

3. What will be the output loss value after one step of training with this code snippet?

import torch
import torch.nn as nn
model = nn.Linear(2, 1)
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)
criterion = nn.MSELoss()

inputs = torch.tensor([[1.0, 2.0]])
target = torch.tensor([[1.0]])

optimizer.zero_grad()
output = model(inputs)
loss = criterion(output, target)
loss.backward()
optimizer.step()

print(round(loss.item(), 3))

medium

A. 0.5

B. 0.25

C. Cannot determine exact value

D. 0.75

Solution

1. Step 1: Understand loss calculation

   Loss depends on random initial weights of model, so exact value varies each run.

2. Step 2: Check code behavior

   Since weights are random, output and loss are not fixed; cannot predict exact loss value.

3. Final Answer:

   Cannot determine exact value -> Option C

4. Quick Check:

   Random init means loss unknown [OK]

Hint: Random weights mean loss value varies; exact output unknown [OK]

Common Mistakes:

• Assuming fixed loss value without checking randomness
• Ignoring model initialization effect
• Confusing loss before and after optimizer step

4. Identify the error in this PyTorch optimizer setup code:

optimizer = torch.optim.Adam(model.parameters, lr=0.001)

medium

A. Adam optimizer does not accept learning rate argument

B. model.parameters should be called as model.parameters()

C. optimizer should be torch.SGD not Adam

D. lr value should be an integer, not float

Solution

1. Step 1: Check method usage

   model.parameters is a method and must be called with parentheses to get parameters.

2. Step 2: Validate other options

   Adam accepts lr argument; SGD is not mandatory; lr can be float.

3. Final Answer:

   model.parameters should be called as model.parameters() -> Option B

4. Quick Check:

   Call parameters() method correctly [OK]

Hint: Always call model.parameters() with parentheses [OK]

Common Mistakes:

• Forgetting parentheses on model.parameters
• Thinking Adam can't take lr argument
• Assuming lr must be integer

5. You want to train a model that converges quickly on noisy data. Which optimizer and learning rate combination is generally best to start with?

hard

A. Adam with lr=0.001

B. SGD with lr=0.1

C. SGD with lr=0.0001

D. Adam with lr=0.1

Solution

1. Step 1: Consider optimizer suitability for noisy data

   Adam adapts learning rates and handles noisy gradients better than SGD.

2. Step 2: Choose appropriate learning rate

   Typical starting lr for Adam is 0.001; too high (0.1) can cause instability.

3. Final Answer:

   Adam with lr=0.001 -> Option A

4. Quick Check:

   Adaptive optimizer + moderate lr = Adam 0.001 [OK]

Hint: Use Adam with 0.001 lr for noisy data to converge fast [OK]

Common Mistakes:

• Using too high learning rate with Adam
• Choosing SGD for noisy data without adaptation
• Confusing learning rate scales