Why regularization controls overfitting in PyTorch - Challenge Your Understanding

Challenge - 5 Problems

🧠 Conceptual

intermediate

Why does L2 regularization reduce overfitting?

Imagine you have a model that learns to predict house prices. You add L2 regularization (weight decay) during training. What is the main reason this helps reduce overfitting?

A It adds random noise to the input data, making the model more robust.

B It increases the learning rate so the model trains faster and avoids overfitting.

C It removes some training examples to reduce the dataset size.

D It forces the model weights to be smaller, making the model simpler and less likely to memorize noise.

💡 Hint

Think about how smaller weights affect the model's ability to fit complex patterns.

L2 regularization adds a penalty for large weights, encouraging the model to keep weights small. Smaller weights mean the model is less complex and less likely to fit noise in the training data, which reduces overfitting.

❓ Predict Output

intermediate

Output of training loss with and without dropout

Consider this PyTorch training loop snippet. What will be the difference in training loss behavior when dropout is enabled versus disabled?

PyTorch

import torch
import torch.nn as nn

class SimpleNet(nn.Module):
    def __init__(self, dropout_rate=0.0):
        super().__init__()
        self.fc1 = nn.Linear(10, 5)
        self.dropout = nn.Dropout(dropout_rate)
        self.fc2 = nn.Linear(5, 1)

    def forward(self, x):
        x = torch.relu(self.fc1(x))
        x = self.dropout(x)
        x = self.fc2(x)
        return x

model = SimpleNet(dropout_rate=0.5)
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)
criterion = nn.MSELoss()

x = torch.randn(20, 10)
y = torch.randn(20, 1)

model.train()
for epoch in range(3):
    optimizer.zero_grad()
    output = model(x)
    loss = criterion(output, y)
    loss.backward()
    optimizer.step()
    print(f"Epoch {epoch+1}, Loss: {loss.item():.4f}")

A Loss increases with dropout because the model cannot learn anything.

B Loss is always zero with dropout enabled because it removes neurons.

C Loss decreases more smoothly and generalizes better with dropout enabled, but may be higher during training.

D Loss is unaffected by dropout during training.

💡 Hint

Dropout randomly disables neurons during training. How does this affect training loss?

Dropout prevents the model from relying too much on any one neuron, which can increase training loss slightly but helps the model generalize better and avoid overfitting.

❓ Hyperparameter

advanced

Choosing the right weight decay value

You are training a neural network with L2 regularization (weight decay). What is the effect of setting the weight decay parameter too high?

A The model weights become too small, causing underfitting and poor training accuracy.

B The model trains faster and achieves better accuracy.

C The model ignores the regularization and overfits the data.

D The model weights become very large, causing instability.

💡 Hint

Think about what happens if the penalty on weights is very strong.

Too high weight decay forces weights to shrink excessively, limiting the model's ability to learn complex patterns, leading to underfitting and poor accuracy.

❓ Metrics

advanced

Interpreting validation loss with and without regularization

You train two models on the same data: one with regularization and one without. Both have similar training loss, but the validation loss of the regularized model is lower. What does this indicate?

A Validation loss is not useful for comparing models.

B The regularized model generalizes better and is less overfitted to training data.

C The model without regularization is better because it fits training data perfectly.

D The regularized model is underfitting and performs worse on unseen data.

💡 Hint

Think about what validation loss tells us about model performance on new data.

Lower validation loss with similar training loss means the regularized model is better at generalizing and avoids overfitting, while the other model likely memorizes training data.

🔧 Debug

expert

Why does adding dropout not reduce overfitting in this code?

Look at this PyTorch model training code. Dropout is added but the model still overfits badly. What is the most likely reason?

PyTorch

import torch
import torch.nn as nn

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(20, 50)
        self.dropout = nn.Dropout(0.5)
        self.fc2 = nn.Linear(50, 1)

    def forward(self, x):
        x = torch.relu(self.fc1(x))
        x = self.dropout(x)
        x = self.fc2(x)
        return x

model = Net()
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)
criterion = nn.MSELoss()

x_train = torch.randn(100, 20)
y_train = torch.randn(100, 1)

for epoch in range(10):
    model.train()  # <--- Changed from model.eval() to model.train()
    optimizer.zero_grad()
    output = model(x_train)
    loss = criterion(output, y_train)
    loss.backward()
    optimizer.step()
    print(f"Epoch {epoch+1}, Loss: {loss.item():.4f}")

A The model is in evaluation mode during training, so dropout is disabled and does not regularize.

B The learning rate is too low, so dropout has no effect.

C The dropout rate is too high, causing the model to ignore inputs.

D The loss function is incorrect for regression tasks.

💡 Hint

Check the mode the model is in during training and how dropout behaves in different modes.

Dropout only works during training mode. Setting model.eval() disables dropout, so the model does not get regularized and overfits.

Practice

1. Why does regularization help prevent overfitting in a PyTorch model?

easy

A. It keeps the model weights small by adding a penalty to the loss.

B. It increases the size of the training dataset automatically.

C. It removes layers from the neural network during training.

D. It speeds up the training process by skipping some data points.

Solution

1. Step 1: Understand what overfitting means

   Overfitting happens when a model learns the training data too well, including noise, causing poor performance on new data.

2. Step 2: Explain how regularization affects model weights

   Regularization adds a penalty to large weights, encouraging smaller weights that generalize better to new data.

3. Final Answer:

   It keeps the model weights small by adding a penalty to the loss. -> Option A

4. Quick Check:

   Regularization = penalty on weights = less overfitting [OK]

Hint: Regularization adds penalty to weights to reduce overfitting [OK]

Common Mistakes:

• Thinking regularization increases data size
• Believing regularization removes layers
• Assuming regularization speeds training

2. Which PyTorch code snippet correctly applies L2 regularization (weight decay) during optimizer setup?

easy

A. optimizer = torch.optim.SGD(model.parameters(), lr=0.01, momentum=0.1)

B. optimizer = torch.optim.SGD(model.parameters(), lr=0.01, dropout=0.1)

C. optimizer = torch.optim.SGD(model.parameters(), lr=0.01, weight_decay=0.1)

D. optimizer = torch.optim.SGD(model.parameters(), lr=0.01, decay=0.1)

Solution

1. Step 1: Identify correct parameter for L2 regularization in PyTorch

   PyTorch uses weight_decay in optimizers to apply L2 regularization.

2. Step 2: Check the code options for correct usage

   Only optimizer = torch.optim.SGD(model.parameters(), lr=0.01, weight_decay=0.1) uses weight_decay=0.1, which is the correct way to add L2 regularization.

3. Final Answer:

   optimizer = torch.optim.SGD(model.parameters(), lr=0.01, weight_decay=0.1) -> Option C

4. Quick Check:

   weight_decay = L2 regularization in PyTorch [OK]

Hint: Use weight_decay param for L2 regularization in PyTorch optimizers [OK]

Common Mistakes:

• Using dropout parameter in optimizer
• Confusing momentum with regularization
• Using decay instead of weight_decay

3. Consider this PyTorch training loop snippet with L2 regularization applied:

optimizer = torch.optim.Adam(model.parameters(), lr=0.001, weight_decay=0.01)
for data, target in dataloader:
    optimizer.zero_grad()
    output = model(data)
    loss = loss_fn(output, target)
    loss.backward()
    optimizer.step()

What effect does the weight_decay=0.01 have during training?

medium

A. It adds a penalty to large weights, helping reduce overfitting.

B. It increases the learning rate by 0.01 each step.

C. It drops 1% of neurons randomly during training.

D. It stops training early when loss is below 0.01.

Solution

1. Step 1: Understand weight_decay in optimizer

   The weight_decay parameter adds L2 regularization, penalizing large weights during training.

2. Step 2: Identify the effect on training

   This penalty helps the model avoid overfitting by keeping weights smaller and more generalizable.

3. Final Answer:

   It adds a penalty to large weights, helping reduce overfitting. -> Option A

4. Quick Check:

   weight_decay = L2 penalty = less overfitting [OK]

Hint: weight_decay adds penalty to weights, not learning rate or dropout [OK]

Common Mistakes:

• Confusing weight_decay with learning rate changes
• Thinking weight_decay is dropout
• Assuming weight_decay controls early stopping

4. You have this PyTorch code snippet intended to apply L2 regularization:

optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
for data, target in dataloader:
    optimizer.zero_grad()
    output = model(data)
    loss = loss_fn(output, target) + 0.01 * torch.sum(model.parameters())
    loss.backward()
    optimizer.step()

What is wrong with this code regarding regularization?

medium

A. It uses SGD optimizer which does not support regularization.

B. It forgets to call optimizer.zero_grad() before backward.

C. It applies regularization after optimizer.step(), so no effect.

D. It incorrectly sums parameters instead of their squares for L2 penalty.

Solution

1. Step 1: Check how L2 regularization is computed

   L2 regularization requires summing the squares of parameters, not just their values.

2. Step 2: Analyze the code's regularization term

   The code sums parameters directly with torch.sum(model.parameters()), which is incorrect for L2 penalty.

3. Final Answer:

   It incorrectly sums parameters instead of their squares for L2 penalty. -> Option D

4. Quick Check:

   L2 penalty = sum of squares, not sum of values [OK]

Hint: L2 regularization sums squares of weights, not weights themselves [OK]

Common Mistakes:

• Summing parameters instead of squared parameters
• Thinking SGD can't use regularization
• Misplacing optimizer.zero_grad() call

5. You train two PyTorch models on the same dataset: Model A uses no regularization, Model B uses L2 regularization with weight_decay=0.05. After training, Model A has training accuracy 98% but test accuracy 70%, while Model B has training accuracy 90% and test accuracy 85%. What explains this difference?

hard

A. Model A's higher training accuracy means it will always perform better on test data.

B. Model B's regularization reduced overfitting by keeping weights smaller, improving test accuracy.

C. Model B used a larger learning rate, causing better generalization.

D. Model A trained longer, so it has better test accuracy.

Solution

1. Step 1: Compare training and test accuracies

   Model A fits training data very well but performs poorly on test data, indicating overfitting.

2. Step 2: Understand effect of L2 regularization on Model B

   Model B has lower training accuracy but better test accuracy because regularization keeps weights smaller, improving generalization.

3. Final Answer:

   Model B's regularization reduced overfitting by keeping weights smaller, improving test accuracy. -> Option B

4. Quick Check:

   Regularization = smaller weights = better test accuracy [OK]

Hint: Better test accuracy with regularization means less overfitting [OK]

Common Mistakes:

• Assuming higher training accuracy means better test accuracy
• Confusing learning rate with regularization effect
• Ignoring the role of weight size in generalization