Optimizers (SGD, Adam) in PyTorch - Model Metrics & Evaluation

Metrics & Evaluation - Optimizers (SGD, Adam)

Which metric matters for Optimizers and WHY

When using optimizers like SGD or Adam, the key metric to watch is the training loss. This tells us how well the model is learning from data step-by-step. A good optimizer helps the loss go down smoothly and quickly.

We also look at validation loss and accuracy to check if the model is improving on new data, not just memorizing training data.

Confusion Matrix Example

While optimizers do not directly affect confusion matrices, the model trained with them produces predictions that can be evaluated with a confusion matrix.

      Confusion Matrix (Example):
      --------------------------
      |         | Pred Pos | Pred Neg |
      |---------|----------|----------|
      | True Pos|   80     |   20     |
      | True Neg|   10     |   90     |
      --------------------------
    

This matrix helps calculate precision and recall, which show how well the optimizer helped the model learn to classify.

Precision vs Recall Tradeoff with Optimizers

Optimizers affect how fast and well a model learns, which impacts precision and recall.

SGD often requires careful tuning and may learn slower, possibly leading to lower recall if training stops early.

Adam adapts learning rates and often reaches better precision and recall faster, but can sometimes overfit if not monitored.

Example: For a spam filter, high precision means fewer good emails marked as spam. Adam might help reach this faster, but SGD can be more stable with tuning.

Good vs Bad Metric Values for Optimizers

Good: Training loss steadily decreases, validation loss decreases or stays stable, accuracy improves, precision and recall balance well.

Bad: Training loss stuck or bouncing, validation loss increasing (overfitting), accuracy not improving, precision or recall very low.

For example, if after many epochs the loss does not improve, the optimizer settings may be wrong.

Common Pitfalls with Optimizer Metrics

• Accuracy Paradox: High accuracy but poor recall or precision means the model is not truly good.
• Data Leakage: If validation data leaks into training, loss and accuracy look better but model fails in real use.
• Overfitting: Training loss goes down but validation loss goes up, optimizer may be too aggressive or learning rate too high.
• Ignoring Learning Rate: Wrong learning rate can cause loss to not improve or diverge.

Self Check

Your model trained with Adam optimizer has 98% accuracy but only 12% recall on fraud cases. Is it good for production?

Answer: No. The low recall means the model misses most fraud cases, which is dangerous. Despite high accuracy, the model is not reliable for fraud detection. You should improve recall, possibly by tuning the optimizer or model.

Key Result

Training loss and validation metrics show how well optimizers like SGD and Adam help the model learn effectively.

Practice

1. Which optimizer in PyTorch adapts the learning rate for each parameter automatically?

easy

A. Adam

B. SGD

C. RMSprop

D. Adagrad

Solution

1. Step 1: Understand optimizer behavior

   Adam optimizer adjusts learning rates for each parameter based on past gradients.

2. Step 2: Compare with SGD

   SGD uses a fixed learning rate for all parameters without adaptation.

3. Final Answer:

   Adam -> Option A

4. Quick Check:

   Adaptive learning rate = Adam [OK]

Hint: Adam adapts learning rates per parameter automatically [OK]

Common Mistakes:

• Confusing SGD as adaptive optimizer
• Thinking RMSprop is the only adaptive optimizer
• Assuming Adagrad is default in PyTorch

2. Which of the following is the correct way to create an SGD optimizer in PyTorch with learning rate 0.01?

easy

A. optimizer = torch.optim.SGD(model, lr=0.01)

B. optimizer = torch.SGD(model.parameters(), learning_rate=0.01)

C. optimizer = torch.optim.Adam(model.parameters(), lr=0.01)

D. optimizer = torch.optim.SGD(model.parameters(), lr=0.01)

Solution

1. Step 1: Check PyTorch optimizer syntax

   Correct syntax uses torch.optim.SGD with model.parameters() and lr argument.

2. Step 2: Identify errors in other options

   The option with torch.SGD(model.parameters(), learning_rate=0.01) uses wrong module and argument name; the option with Adam uses wrong optimizer; the option passing model instead of parameters is incorrect.

3. Final Answer:

   optimizer = torch.optim.SGD(model.parameters(), lr=0.01) -> Option D

4. Quick Check:

   Correct syntax = optimizer = torch.optim.SGD(model.parameters(), lr=0.01) [OK]

Hint: Use torch.optim.SGD with model.parameters() and lr= value [OK]

Common Mistakes:

• Passing model instead of model.parameters()
• Using wrong argument name like learning_rate
• Confusing Adam with SGD

3. What will be the output loss value after one step of training with this code snippet?

import torch
import torch.nn as nn
model = nn.Linear(2, 1)
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)
criterion = nn.MSELoss()

inputs = torch.tensor([[1.0, 2.0]])
target = torch.tensor([[1.0]])

optimizer.zero_grad()
output = model(inputs)
loss = criterion(output, target)
loss.backward()
optimizer.step()

print(round(loss.item(), 3))

medium

A. 0.5

B. 0.25

C. Cannot determine exact value

D. 0.75

Solution

1. Step 1: Understand loss calculation

   Loss depends on random initial weights of model, so exact value varies each run.

2. Step 2: Check code behavior

   Since weights are random, output and loss are not fixed; cannot predict exact loss value.

3. Final Answer:

   Cannot determine exact value -> Option C

4. Quick Check:

   Random init means loss unknown [OK]

Hint: Random weights mean loss value varies; exact output unknown [OK]

Common Mistakes:

• Assuming fixed loss value without checking randomness
• Ignoring model initialization effect
• Confusing loss before and after optimizer step

4. Identify the error in this PyTorch optimizer setup code:

optimizer = torch.optim.Adam(model.parameters, lr=0.001)

medium

A. Adam optimizer does not accept learning rate argument

B. model.parameters should be called as model.parameters()

C. optimizer should be torch.SGD not Adam

D. lr value should be an integer, not float

Solution

1. Step 1: Check method usage

   model.parameters is a method and must be called with parentheses to get parameters.

2. Step 2: Validate other options

   Adam accepts lr argument; SGD is not mandatory; lr can be float.

3. Final Answer:

   model.parameters should be called as model.parameters() -> Option B

4. Quick Check:

   Call parameters() method correctly [OK]

Hint: Always call model.parameters() with parentheses [OK]

Common Mistakes:

• Forgetting parentheses on model.parameters
• Thinking Adam can't take lr argument
• Assuming lr must be integer

5. You want to train a model that converges quickly on noisy data. Which optimizer and learning rate combination is generally best to start with?

hard

A. Adam with lr=0.001

B. SGD with lr=0.1

C. SGD with lr=0.0001

D. Adam with lr=0.1

Solution

1. Step 1: Consider optimizer suitability for noisy data

   Adam adapts learning rates and handles noisy gradients better than SGD.

2. Step 2: Choose appropriate learning rate

   Typical starting lr for Adam is 0.001; too high (0.1) can cause instability.

3. Final Answer:

   Adam with lr=0.001 -> Option A

4. Quick Check:

   Adaptive optimizer + moderate lr = Adam 0.001 [OK]

Hint: Use Adam with 0.001 lr for noisy data to converge fast [OK]

Common Mistakes:

• Using too high learning rate with Adam
• Choosing SGD for noisy data without adaptation
• Confusing learning rate scales