Sequential model shortcut in PyTorch - ML Experiment: Train & Evaluate

Experiment - Sequential model shortcut

Problem:You have a simple neural network built with PyTorch's Sequential model to classify digits from the MNIST dataset. The model currently achieves 98% training accuracy but only 85% validation accuracy.

Current Metrics:Training accuracy: 98%, Validation accuracy: 85%, Training loss: 0.05, Validation loss: 0.35

Issue:The model is overfitting: training accuracy is very high but validation accuracy is much lower.

Your Task

Reduce overfitting by improving validation accuracy to at least 90% while keeping training accuracy below 95%.

You must keep using PyTorch Sequential model.

You cannot change the dataset or increase its size.

You can only modify the model architecture or training hyperparameters.

Hint 1

Hint 2

Hint 3

Solution

PyTorch

import torch
import torch.nn as nn
import torch.optim as optim
from torchvision import datasets, transforms
from torch.utils.data import DataLoader

# Data preparation
transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.1307,), (0.3081,))
])

train_dataset = datasets.MNIST(root='./data', train=True, download=True, transform=transform)
val_dataset = datasets.MNIST(root='./data', train=False, download=True, transform=transform)

train_loader = DataLoader(train_dataset, batch_size=64, shuffle=True)
val_loader = DataLoader(val_dataset, batch_size=1000, shuffle=False)

# Define model with dropout and batch normalization
model = nn.Sequential(
    nn.Flatten(),
    nn.Linear(28*28, 256),
    nn.BatchNorm1d(256),
    nn.ReLU(),
    nn.Dropout(0.3),
    nn.Linear(256, 128),
    nn.BatchNorm1d(128),
    nn.ReLU(),
    nn.Dropout(0.3),
    nn.Linear(128, 10)
)

# Loss and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)

# Training loop
for epoch in range(10):
    model.train()
    total_loss = 0
    correct = 0
    for data, target in train_loader:
        optimizer.zero_grad()
        output = model(data)
        loss = criterion(output, target)
        loss.backward()
        optimizer.step()
        total_loss += loss.item() * data.size(0)
        pred = output.argmax(dim=1)
        correct += pred.eq(target).sum().item()
    train_loss = total_loss / len(train_loader.dataset)
    train_acc = correct / len(train_loader.dataset) * 100

    model.eval()
    val_loss = 0
    val_correct = 0
    with torch.no_grad():
        for data, target in val_loader:
            output = model(data)
            loss = criterion(output, target)
            val_loss += loss.item() * data.size(0)
            pred = output.argmax(dim=1)
            val_correct += pred.eq(target).sum().item()
    val_loss /= len(val_loader.dataset)
    val_acc = val_correct / len(val_loader.dataset) * 100

    print(f'Epoch {epoch+1}: Train Loss={train_loss:.4f}, Train Acc={train_acc:.2f}%, Val Loss={val_loss:.4f}, Val Acc={val_acc:.2f}%')

Added nn.Dropout layers with 0.3 dropout rate after activation layers to reduce overfitting.

Added nn.BatchNorm1d layers after linear layers to stabilize and speed up training.

Kept learning rate at 0.001 and trained for 10 epochs.

Results Interpretation

Before: Training accuracy: 98%, Validation accuracy: 85%, Training loss: 0.05, Validation loss: 0.35

After: Training accuracy: 93%, Validation accuracy: 91%, Training loss: 0.15, Validation loss: 0.25

Adding dropout and batch normalization helps reduce overfitting by preventing the model from relying too much on training data details. This improves validation accuracy and generalization.

Bonus Experiment

Try replacing the dropout layers with L2 weight regularization (weight decay) in the optimizer and compare results.

💡 Hint

Set weight_decay parameter in Adam optimizer to a small value like 0.0005 and remove dropout layers.

Practice

1. What is the main advantage of using torch.nn.Sequential in PyTorch?

easy

A. It allows stacking layers quickly without writing a forward method.

B. It automatically tunes hyperparameters during training.

C. It visualizes the model architecture graphically.

D. It replaces the need for a loss function.

Solution

1. Step 1: Understand what Sequential does

   torch.nn.Sequential stacks layers in order and handles the forward pass automatically.

2. Step 2: Compare other options

   Hyperparameter tuning, visualization, and replacing the loss function are not features of Sequential.

3. Final Answer:

   It allows stacking layers quickly without writing a forward method. -> Option A

4. Quick Check:

   Sequential = quick layer stacking [OK]

Hint: Sequential stacks layers; no manual forward needed [OK]

Common Mistakes:

• Thinking Sequential tunes hyperparameters
• Confusing Sequential with visualization tools
• Assuming Sequential replaces loss functions

2. Which of the following is the correct way to create a simple feedforward network with two linear layers using torch.nn.Sequential?

easy

A. model = nn.Sequential(nn.Linear(10, 5) nn.ReLU() nn.Linear(5, 2))

B. model = nn.Sequential([nn.Linear(10, 5), nn.ReLU(), nn.Linear(5, 2)])

C. model = nn.Sequential(nn.Linear(10, 5), nn.ReLU(), nn.Linear(5, 2))

D. model = nn.Sequential{nn.Linear(10, 5), nn.ReLU(), nn.Linear(5, 2)}

Solution

1. Step 1: Check correct syntax for Sequential

   nn.Sequential takes layers as separate arguments, separated by commas and enclosed in parentheses.

2. Step 2: Identify syntax errors in other options

   A misses commas between layers causing a syntax error, B uses a list which is invalid here, C uses curly braces which is wrong.

3. Final Answer:

   model = nn.Sequential(nn.Linear(10, 5), nn.ReLU(), nn.Linear(5, 2)) -> Option C

4. Quick Check:

   Sequential layers separated by commas in parentheses [OK]

Hint: Use commas between layers inside parentheses [OK]

Common Mistakes:

• Using square brackets inside Sequential
• Using curly braces instead of parentheses
• Forgetting commas between layers

3. Given the following code, what will be the output shape of output?

import torch
import torch.nn as nn
model = nn.Sequential(
    nn.Linear(4, 3),
    nn.ReLU(),
    nn.Linear(3, 2)
)
input = torch.randn(5, 4)
output = model(input)
print(output.shape)

medium

A. torch.Size([4, 2])

B. torch.Size([5, 3])

C. torch.Size([5, 4])

D. torch.Size([5, 2])

Solution

1. Step 1: Analyze input and first layer

   Input shape is (5,4). First Linear layer maps 4 features to 3, so output shape becomes (5,3).

2. Step 2: Analyze second Linear layer output

   Second Linear layer maps 3 features to 2, so final output shape is (5,2).

3. Final Answer:

   torch.Size([5, 2]) -> Option D

4. Quick Check:

   Input batch 5, final features 2 [OK]

Hint: Track input batch size and final layer output features [OK]

Common Mistakes:

• Confusing input and output feature sizes
• Ignoring batch dimension
• Assuming ReLU changes shape

4. What is wrong with this PyTorch Sequential model code?

import torch.nn as nn
model = nn.Sequential(
    nn.Linear(10, 5),
    nn.ReLU,
    nn.Linear(5, 2)
)

medium

A. Missing parentheses after nn.ReLU causing a type error.

B. Linear layers have wrong input-output sizes.

C. Sequential cannot contain activation functions.

D. Model should be defined using nn.Module subclass.

Solution

1. Step 1: Check usage of nn.ReLU

   nn.ReLU is a class; it must be called with parentheses to create an instance.

2. Step 2: Verify other options

   Linear layers sizes are compatible, Sequential supports activations, and subclassing is optional.

3. Final Answer:

   Missing parentheses after nn.ReLU causing a type error. -> Option A

4. Quick Check:

   Activation needs parentheses () [OK]

Hint: Always add () after activation layers [OK]

Common Mistakes:

• Forgetting () after activation functions
• Thinking Sequential disallows activations
• Believing subclassing is mandatory

5. You want to build a neural network with three layers: input size 8, hidden size 4, and output size 1. You also want to add dropout after the first layer and use ReLU activations. Which nn.Sequential model correctly implements this?

hard

A. nn.Sequential(nn.Linear(8,4), nn.ReLU, nn.Dropout(0.5), nn.Linear(4,1))

B. nn.Sequential(nn.Linear(8,4), nn.Dropout(0.5), nn.ReLU(), nn.Linear(4,1))

C. nn.Sequential(nn.Linear(8,4), nn.ReLU(), nn.Linear(4,1), nn.Dropout(0.5))

D. nn.Sequential(nn.Dropout(0.5), nn.Linear(8,4), nn.ReLU(), nn.Linear(4,1))

Solution

1. Step 1: Check layer order and activations

   Dropout should come after the first Linear layer and before ReLU to randomly drop activations during training.

2. Step 2: Verify correct usage of activations and dropout

   nn.Sequential(nn.Linear(8,4), nn.Dropout(0.5), nn.ReLU(), nn.Linear(4,1)) correctly uses nn.ReLU() with parentheses and places dropout before ReLU. Other options either miss parentheses or misplace dropout.

3. Final Answer:

   nn.Sequential(nn.Linear(8,4), nn.Dropout(0.5), nn.ReLU(), nn.Linear(4,1)) -> Option B

4. Quick Check:

   Dropout after Linear, before ReLU, with () [OK]

Hint: Dropout after Linear, before ReLU, use () for activations [OK]

Common Mistakes:

• Placing dropout after output layer
• Forgetting parentheses on ReLU
• Putting dropout before first Linear layer