Weight decay (L2 regularization) in PyTorch - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

What is the main purpose of weight decay (L2 regularization) in training neural networks?

Weight decay is often used during training of neural networks. What does it mainly help with?

A It speeds up the training by increasing the learning rate automatically.

B It helps the model memorize the training data perfectly.

C It prevents the model from overfitting by penalizing large weights.

D It changes the activation functions to nonlinear ones.

💡 Hint

Think about what happens when weights become very large and how that affects generalization.

Weight decay adds a penalty to large weights, encouraging the model to keep weights small. This reduces overfitting and helps the model generalize better.

❓ Predict Output

intermediate

What is the output of this PyTorch training loop snippet with weight decay?

Consider this PyTorch code snippet training a simple linear model with weight decay. What will be printed?

PyTorch

import torch
import torch.nn as nn
import torch.optim as optim

model = nn.Linear(1, 1)
optimizer = optim.SGD(model.parameters(), lr=0.1, weight_decay=0.1)

x = torch.tensor([[1.0]])
y = torch.tensor([[2.0]])

criterion = nn.MSELoss()

for _ in range(1):
    optimizer.zero_grad()
    output = model(x)
    loss = criterion(output, y)
    loss.backward()
    optimizer.step()

print(round(loss.item(), 3))

A 0.0

B 0.25

C 0.81

D 1.0

💡 Hint

Think about the initial random weights and one step of gradient descent with weight decay.

The initial loss is roughly the squared difference between output and target. Weight decay slightly affects the update but loss after one step remains close to 0.25.

❓ Model Choice

advanced

Which optimizer setup correctly applies weight decay only to weights but not biases in PyTorch?

You want to apply weight decay only to the weights of a neural network, not to the bias terms. Which optimizer setup below does this correctly?

A

optimizer = optim.Adam([
    {'params': [p for n, p in model.named_parameters() if 'bias' not in n], 'weight_decay': 0.01},
    {'params': [p for n, p in model.named_parameters() if 'bias' in n], 'weight_decay': 0.0}
], lr=0.001)

B optimizer = optim.Adam(model.parameters(), lr=0.001, weight_decay=0.01)

C optimizer = optim.Adam(model.parameters(), lr=0.001)

D

optimizer = optim.Adam([
    {'params': model.parameters(), 'weight_decay': 0.0}
], lr=0.001)

💡 Hint

Think about how to separate parameters by name and assign different weight decay values.

Option A correctly separates bias parameters and applies zero weight decay to them, while applying weight decay to other parameters.

❓ Hyperparameter

advanced

What is the effect of increasing the weight decay hyperparameter too much during training?

If you increase the weight decay value excessively, what is the most likely effect on the model's training?

A The model will ignore the training data and memorize the validation set.

B The model weights will become very large and unstable.

C The model will train faster and achieve higher accuracy.

D The model weights will shrink too much, causing underfitting and poor performance.

💡 Hint

Consider what happens if the penalty on weights is very strong.

Too much weight decay forces weights to be very small, limiting the model's ability to learn patterns, causing underfitting.

🔧 Debug

expert

Why does this PyTorch training code not apply weight decay as expected?

Look at this PyTorch code snippet. The user expects weight decay to be applied, but the model's weights do not shrink. What is the bug?

PyTorch

import torch
import torch.nn as nn
import torch.optim as optim

model = nn.Linear(2, 1)
optimizer = optim.SGD(model.parameters(), lr=0.1, weight_decay=0.01)

x = torch.tensor([[1.0, 2.0]])
y = torch.tensor([[1.0]])

criterion = nn.MSELoss()

for _ in range(10):
    optimizer.zero_grad()
    output = model(x)
    loss = criterion(output, y)
    loss.backward()
    optimizer.step()

print(model.weight)

A Weight decay is not applied because the model's bias parameters are updated but weights are not.

B Weight decay is not applied because the model's weights are frozen by default.

C Weight decay is not applied because the optimizer is created before model parameters are initialized.

D The weight_decay parameter is ignored because the learning rate is too high.

💡 Hint

Check which parameters are updated and how weight decay affects weights vs biases.

Weight decay applies only to weights, not biases. If only biases are updated or weights remain unchanged, weight decay effect is not visible.