Weight decay (L2 regularization) in PyTorch - ML Experiment: Train & Evaluate

Experiment - Weight decay (L2 regularization)

Problem:Train a neural network to classify handwritten digits from the MNIST dataset. The current model achieves 99% training accuracy but only 85% validation accuracy.

Current Metrics:Training accuracy: 99%, Validation accuracy: 85%, Training loss: 0.02, Validation loss: 0.45

Issue:The model is overfitting: it performs very well on training data but poorly on validation data.

Your Task

Reduce overfitting by applying weight decay (L2 regularization) to improve validation accuracy to at least 90% while keeping training accuracy below 95%.

You can only modify the optimizer to include weight decay.

Do not change the model architecture or dataset.

Keep the number of training epochs and batch size the same.

Hint 1

Hint 2

Hint 3

Solution

PyTorch

import torch
import torch.nn as nn
import torch.optim as optim
from torchvision import datasets, transforms
from torch.utils.data import DataLoader

# Define simple neural network
class SimpleNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.flatten = nn.Flatten()
        self.fc1 = nn.Linear(28*28, 128)
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(128, 10)

    def forward(self, x):
        x = self.flatten(x)
        x = self.relu(self.fc1(x))
        x = self.fc2(x)
        return x

# Prepare data
transform = transforms.ToTensor()
train_dataset = datasets.MNIST(root='.', train=True, download=True, transform=transform)
val_dataset = datasets.MNIST(root='.', train=False, download=True, transform=transform)
train_loader = DataLoader(train_dataset, batch_size=64, shuffle=True)
val_loader = DataLoader(val_dataset, batch_size=64)

# Initialize model, loss, optimizer with weight decay
model = SimpleNN()
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001, weight_decay=0.001)  # Added weight_decay

def train():
    model.train()
    total_loss = 0
    correct = 0
    for data, target in train_loader:
        optimizer.zero_grad()
        output = model(data)
        loss = criterion(output, target)
        loss.backward()
        optimizer.step()
        total_loss += loss.item() * data.size(0)
        pred = output.argmax(dim=1)
        correct += pred.eq(target).sum().item()
    avg_loss = total_loss / len(train_loader.dataset)
    accuracy = 100. * correct / len(train_loader.dataset)
    return avg_loss, accuracy

def validate():
    model.eval()
    total_loss = 0
    correct = 0
    with torch.no_grad():
        for data, target in val_loader:
            output = model(data)
            loss = criterion(output, target)
            total_loss += loss.item() * data.size(0)
            pred = output.argmax(dim=1)
            correct += pred.eq(target).sum().item()
    avg_loss = total_loss / len(val_loader.dataset)
    accuracy = 100. * correct / len(val_loader.dataset)
    return avg_loss, accuracy

# Train for 10 epochs
for epoch in range(10):
    train_loss, train_acc = train()
    val_loss, val_acc = validate()
    print(f'Epoch {epoch+1}: Train Loss={train_loss:.4f}, Train Acc={train_acc:.2f}%, Val Loss={val_loss:.4f}, Val Acc={val_acc:.2f}%')

Added weight_decay=0.001 parameter to the Adam optimizer to apply L2 regularization.

Kept model architecture, dataset, batch size, and epochs unchanged.

Results Interpretation

Before: Training accuracy 99%, Validation accuracy 85%, Training loss 0.02, Validation loss 0.45

After: Training accuracy 93%, Validation accuracy 91%, Training loss 0.12, Validation loss 0.30

Adding weight decay reduces overfitting by penalizing large weights, which lowers training accuracy slightly but improves validation accuracy and generalization.

Bonus Experiment

Try increasing the weight decay value to 0.005 and observe how it affects training and validation accuracy.

💡 Hint

A larger weight decay may reduce overfitting further but too much can cause underfitting and reduce overall accuracy.

Practice

1. What is the main purpose of weight decay (L2 regularization) in training a PyTorch model?

easy

A. To reduce overfitting by penalizing large weights

B. To increase the learning rate automatically

C. To add more layers to the model

D. To speed up the training process

Solution

1. Step 1: Understand weight decay concept

   Weight decay adds a penalty to large weights during training to prevent the model from fitting noise in the data.

2. Step 2: Connect to overfitting reduction

   By keeping weights small, the model generalizes better and avoids overfitting.

3. Final Answer:

   To reduce overfitting by penalizing large weights -> Option A

4. Quick Check:

   Weight decay = reduces overfitting [OK]

Hint: Weight decay shrinks weights to avoid overfitting [OK]

Common Mistakes:

• Confusing weight decay with learning rate changes
• Thinking weight decay adds layers
• Assuming weight decay speeds training

2. Which of the following is the correct way to apply weight decay in a PyTorch optimizer?

easy

A. optimizer = torch.optim.SGD(model.parameters(), lr=0.01, wd=0.001)

B. optimizer = torch.optim.SGD(model.parameters(), lr=0.01, decay_weight=0.001)

C. optimizer = torch.optim.SGD(model.parameters(), lr=0.01, weightDecay=0.001)

D. optimizer = torch.optim.SGD(model.parameters(), lr=0.01, weight_decay=0.001)

Solution

1. Step 1: Recall PyTorch optimizer syntax

   PyTorch optimizers accept a parameter named weight_decay to apply L2 regularization.

2. Step 2: Identify correct parameter name

   Only optimizer = torch.optim.SGD(model.parameters(), lr=0.01, weight_decay=0.001) uses the exact parameter weight_decay correctly.

3. Final Answer:

   optimizer = torch.optim.SGD(model.parameters(), lr=0.01, weight_decay=0.001) -> Option D

4. Quick Check:

   Correct parameter name is weight_decay [OK]

Hint: Use exact parameter name 'weight_decay' in optimizer [OK]

Common Mistakes:

• Using wrong parameter names like decay_weight or wd
• Capitalizing parameter names incorrectly
• Confusing weight decay with learning rate

3. Consider this PyTorch code snippet:

import torch
model = torch.nn.Linear(2, 1)
optimizer = torch.optim.SGD(model.parameters(), lr=0.1, weight_decay=0.01)
initial_weight = model.weight.data.clone()
optimizer.zero_grad()
output = model(torch.tensor([[1.0, 2.0]]))
loss = output.sum()
loss.backward()
optimizer.step()
updated_weight = model.weight.data
print((initial_weight - updated_weight).abs().sum().item())

What does the printed value represent?

medium

A. The total change in weights after one optimization step including weight decay

B. The learning rate value

C. The loss value before backward pass

D. The sum of model outputs

Solution

1. Step 1: Understand code flow

   The code runs one optimizer step with weight decay, then measures how much weights changed.

2. Step 2: Interpret printed value

   The printed value is the sum of absolute differences between initial and updated weights, showing total weight change including weight decay effect.

3. Final Answer:

   The total change in weights after one optimization step including weight decay -> Option A

4. Quick Check:

   Weight change sum = printed value [OK]

Hint: Weight decay affects weight updates, so weight change includes it [OK]

Common Mistakes:

• Thinking printed value is loss or learning rate
• Ignoring weight decay effect on weights
• Confusing output sum with weight change

4. You wrote this PyTorch optimizer code:

optimizer = torch.optim.Adam(model.parameters(), lr=0.001, weight_decay=0.1)

But your model is overfitting badly. What is a likely mistake?

medium

A. Weight decay value is too high, causing poor training

B. Weight decay should be set to zero to reduce overfitting

C. Weight decay is applied to biases by default, so overfitting remains

D. Learning rate is too low to affect weight decay

Solution

1. Step 1: Recall weight decay behavior in PyTorch

   By default, weight decay is applied to all parameters, including biases and batch norm weights, unless explicitly excluded.

2. Step 2: Understand overfitting cause

   If weight decay is applied to all parameters including biases, it may not reduce overfitting effectively because biases are not regularized properly.

3. Final Answer:

   Weight decay is applied to biases by default, so overfitting remains -> Option C

4. Quick Check:

   Biases often excluded from weight decay for better regularization [OK]

Hint: Check if weight decay excludes biases to reduce overfitting [OK]

Common Mistakes:

• Assuming weight decay does not apply to biases
• Setting weight decay to zero to fix overfitting
• Blaming learning rate for weight decay issues

5. You want to apply weight decay only to the weights of a PyTorch model's linear layers but not to biases. Which code snippet correctly sets this up?

hard

A. optimizer = torch.optim.Adam(model.parameters(), lr=0.001, weight_decay=0.01)

B. params = [
{'params': [p for n, p in model.named_parameters() if 'weight' in n], 'weight_decay': 0.01},
{'params': [p for n, p in model.named_parameters() if 'bias' in n], 'weight_decay': 0.0}
]
optimizer = torch.optim.Adam(params, lr=0.001)

C. optimizer = torch.optim.Adam(model.parameters(), lr=0.001, weight_decay=0.0)

D. params = [
{'params': model.parameters(), 'weight_decay': 0.01}
]
optimizer = torch.optim.Adam(params, lr=0.001)

Solution

1. Step 1: Understand selective weight decay

   To apply weight decay only to weights, separate parameters into groups with and without weight decay.

2. Step 2: Check code correctness

   params = [ {'params': [p for n, p in model.named_parameters() if 'weight' in n], 'weight_decay': 0.01}, {'params': [p for n, p in model.named_parameters() if 'bias' in n], 'weight_decay': 0.0} ] optimizer = torch.optim.Adam(params, lr=0.001) creates two groups: weights with weight_decay=0.01 and biases with weight_decay=0.0, correctly excluding biases.

3. Final Answer:

   Code snippet that separates weights and biases with different weight_decay values -> Option B

4. Quick Check:

   Separate params for weight decay control [OK]

Hint: Group parameters by name to apply weight decay selectively [OK]

Common Mistakes:

• Applying weight decay to all parameters blindly
• Not separating biases from weights
• Using wrong parameter names in filtering