Optimizers (SGD, Adam) in PyTorch - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Difference between SGD and Adam optimizers

Which statement correctly describes a key difference between SGD and Adam optimizers?

A SGD uses adaptive learning rates based on gradient variance, while Adam uses a constant learning rate.

B SGD uses momentum by default, while Adam does not use momentum at all.

C Adam requires manual tuning of learning rate decay, but SGD automatically adjusts learning rate during training.

D Adam adapts the learning rate for each parameter individually using estimates of first and second moments, while SGD uses a fixed learning rate for all parameters.

💡 Hint

Think about how each optimizer handles learning rates for different parameters.

Adam optimizer calculates adaptive learning rates for each parameter using estimates of first (mean) and second (variance) moments of gradients. SGD applies the same learning rate to all parameters unless manually changed.

❓ Predict Output

intermediate

Output of training loss with SGD vs Adam

Given the following PyTorch training loop snippet, what will be the printed loss values after one update step using SGD and Adam optimizers respectively?

PyTorch

import torch
import torch.nn as nn
import torch.optim as optim

model = nn.Linear(2, 1)
criterion = nn.MSELoss()

x = torch.tensor([[1.0, 2.0]])
y = torch.tensor([[1.0]])

# Using SGD optimizer
optimizer_sgd = optim.SGD(model.parameters(), lr=0.1)

# Forward pass
output = model(x)
loss = criterion(output, y)

# Backward and optimize
optimizer_sgd.zero_grad()
loss.backward()
optimizer_sgd.step()

loss_sgd = criterion(model(x), y).item()

# Reset model weights
model = nn.Linear(2, 1)
optimizer_adam = optim.Adam(model.parameters(), lr=0.1)

output = model(x)
loss = criterion(output, y)
optimizer_adam.zero_grad()
loss.backward()
optimizer_adam.step()
loss_adam = criterion(model(x), y).item()

print(round(loss_sgd, 3), round(loss_adam, 3))

A Both losses decrease but Adam's loss decreases more after one step.

B Both losses remain exactly the same after one step.

C SGD loss decreases, but Adam loss increases after one step.

D Adam loss decreases, but SGD loss increases after one step.

💡 Hint

Adam usually converges faster due to adaptive learning rates.

After one update step, both optimizers reduce the loss, but Adam typically reduces it more because it adapts learning rates per parameter.

❓ Hyperparameter

advanced

Choosing learning rate for Adam optimizer

Which learning rate value is generally recommended as a good starting point for the Adam optimizer in PyTorch?

A 0.01

B 0.001

C 0.1

D 1.0

💡 Hint

Adam usually requires smaller learning rates than SGD.

Adam typically works well with a learning rate around 0.001 as a default starting point.

❓ Metrics

advanced

Effect of optimizer on training accuracy

You train the same neural network on a classification task using SGD and Adam optimizers with the same learning rate and number of epochs. Which outcome is most likely regarding training accuracy?

A SGD always achieves higher training accuracy than Adam.

B Both optimizers achieve exactly the same training accuracy at all times.

C Adam achieves higher training accuracy faster than SGD.

D Adam achieves lower training accuracy than SGD due to overfitting.

💡 Hint

Consider how adaptive learning rates affect convergence speed.

Adam usually converges faster and reaches higher accuracy earlier because it adapts learning rates per parameter.

🔧 Debug

expert

Identifying error in optimizer usage

What error will this PyTorch code raise when trying to train a model with Adam optimizer?

PyTorch

import torch
import torch.nn as nn
import torch.optim as optim

model = nn.Linear(3, 1)
optimizer = optim.Adam(model.parameters(), lr=0.01)

x = torch.randn(5, 3)
y = torch.randn(5, 1)

criterion = nn.MSELoss()

optimizer.zero_grad()
output = model(x)
loss = criterion(output, y)

loss.backward()
optimizer.step()

A No error; code runs correctly.

B RuntimeError: optimizer.zero_grad() called after loss.backward() instead of before.

C RuntimeError: Trying to backward through the graph a second time without retaining it.

D TypeError: optimizer.step() missing required positional argument.

💡 Hint

Check the order of zero_grad and backward calls.

optimizer.zero_grad() should be called before loss.backward() to clear old gradients. Calling zero_grad after backward causes incorrect gradient accumulation and runtime error.

Practice

1. Which optimizer in PyTorch adapts the learning rate for each parameter automatically?

easy

A. Adam

B. SGD

C. RMSprop

D. Adagrad

Solution

1. Step 1: Understand optimizer behavior

   Adam optimizer adjusts learning rates for each parameter based on past gradients.

2. Step 2: Compare with SGD

   SGD uses a fixed learning rate for all parameters without adaptation.

3. Final Answer:

   Adam -> Option A

4. Quick Check:

   Adaptive learning rate = Adam [OK]

Hint: Adam adapts learning rates per parameter automatically [OK]

Common Mistakes:

• Confusing SGD as adaptive optimizer
• Thinking RMSprop is the only adaptive optimizer
• Assuming Adagrad is default in PyTorch

2. Which of the following is the correct way to create an SGD optimizer in PyTorch with learning rate 0.01?

easy

A. optimizer = torch.optim.SGD(model, lr=0.01)

B. optimizer = torch.SGD(model.parameters(), learning_rate=0.01)

C. optimizer = torch.optim.Adam(model.parameters(), lr=0.01)

D. optimizer = torch.optim.SGD(model.parameters(), lr=0.01)

Solution

1. Step 1: Check PyTorch optimizer syntax

   Correct syntax uses torch.optim.SGD with model.parameters() and lr argument.

2. Step 2: Identify errors in other options

   The option with torch.SGD(model.parameters(), learning_rate=0.01) uses wrong module and argument name; the option with Adam uses wrong optimizer; the option passing model instead of parameters is incorrect.

3. Final Answer:

   optimizer = torch.optim.SGD(model.parameters(), lr=0.01) -> Option D

4. Quick Check:

   Correct syntax = optimizer = torch.optim.SGD(model.parameters(), lr=0.01) [OK]

Hint: Use torch.optim.SGD with model.parameters() and lr= value [OK]

Common Mistakes:

• Passing model instead of model.parameters()
• Using wrong argument name like learning_rate
• Confusing Adam with SGD

3. What will be the output loss value after one step of training with this code snippet?

import torch
import torch.nn as nn
model = nn.Linear(2, 1)
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)
criterion = nn.MSELoss()

inputs = torch.tensor([[1.0, 2.0]])
target = torch.tensor([[1.0]])

optimizer.zero_grad()
output = model(inputs)
loss = criterion(output, target)
loss.backward()
optimizer.step()

print(round(loss.item(), 3))

medium

A. 0.5

B. 0.25

C. Cannot determine exact value

D. 0.75

Solution

1. Step 1: Understand loss calculation

   Loss depends on random initial weights of model, so exact value varies each run.

2. Step 2: Check code behavior

   Since weights are random, output and loss are not fixed; cannot predict exact loss value.

3. Final Answer:

   Cannot determine exact value -> Option C

4. Quick Check:

   Random init means loss unknown [OK]

Hint: Random weights mean loss value varies; exact output unknown [OK]

Common Mistakes:

• Assuming fixed loss value without checking randomness
• Ignoring model initialization effect
• Confusing loss before and after optimizer step

4. Identify the error in this PyTorch optimizer setup code:

optimizer = torch.optim.Adam(model.parameters, lr=0.001)

medium

A. Adam optimizer does not accept learning rate argument

B. model.parameters should be called as model.parameters()

C. optimizer should be torch.SGD not Adam

D. lr value should be an integer, not float

Solution

1. Step 1: Check method usage

   model.parameters is a method and must be called with parentheses to get parameters.

2. Step 2: Validate other options

   Adam accepts lr argument; SGD is not mandatory; lr can be float.

3. Final Answer:

   model.parameters should be called as model.parameters() -> Option B

4. Quick Check:

   Call parameters() method correctly [OK]

Hint: Always call model.parameters() with parentheses [OK]

Common Mistakes:

• Forgetting parentheses on model.parameters
• Thinking Adam can't take lr argument
• Assuming lr must be integer

5. You want to train a model that converges quickly on noisy data. Which optimizer and learning rate combination is generally best to start with?

hard

A. Adam with lr=0.001

B. SGD with lr=0.1

C. SGD with lr=0.0001

D. Adam with lr=0.1

Solution

1. Step 1: Consider optimizer suitability for noisy data

   Adam adapts learning rates and handles noisy gradients better than SGD.

2. Step 2: Choose appropriate learning rate

   Typical starting lr for Adam is 0.001; too high (0.1) can cause instability.

3. Final Answer:

   Adam with lr=0.001 -> Option A

4. Quick Check:

   Adaptive optimizer + moderate lr = Adam 0.001 [OK]

Hint: Use Adam with 0.001 lr for noisy data to converge fast [OK]

Common Mistakes:

• Using too high learning rate with Adam
• Choosing SGD for noisy data without adaptation
• Confusing learning rate scales