forward method in PyTorch - ML Experiment: Train & Evaluate

Experiment - forward method

Problem:You have built a simple neural network in PyTorch but the model's forward method is not correctly implemented. As a result, the model does not produce valid predictions and training metrics are poor.

Current Metrics:Training loss: 2.3, Training accuracy: 10%, Validation loss: 2.3, Validation accuracy: 10%

Issue:The forward method does not correctly pass input through the layers, causing the model to output random predictions and fail to learn.

Your Task

Fix the forward method so that the input data correctly flows through the network layers, improving training and validation accuracy to above 80%.

Do not change the model architecture or layer definitions.

Only modify the forward method implementation.

Hint 1

Hint 2

Hint 3

Solution

PyTorch

import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader, TensorDataset

# Define a simple neural network
class SimpleNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(28*28, 128)
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(128, 10)

    def forward(self, x):
        x = x.view(x.size(0), -1)  # Flatten input
        x = self.fc1(x)
        x = self.relu(x)
        x = self.fc2(x)
        return x

# Create dummy data (e.g., 100 samples of 28x28 images)
X_train = torch.randn(100, 1, 28, 28)
y_train = torch.randint(0, 10, (100,))

train_dataset = TensorDataset(X_train, y_train)
train_loader = DataLoader(train_dataset, batch_size=10)

# Initialize model, loss, optimizer
model = SimpleNet()
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)

# Training loop
model.train()
for epoch in range(5):
    total_loss = 0
    correct = 0
    total = 0
    for inputs, labels in train_loader:
        optimizer.zero_grad()
        outputs = model(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        total_loss += loss.item() * inputs.size(0)
        _, predicted = torch.max(outputs, 1)
        correct += (predicted == labels).sum().item()
        total += labels.size(0)

    avg_loss = total_loss / total
    accuracy = correct / total * 100
    print(f"Epoch {epoch+1}: Loss={avg_loss:.4f}, Accuracy={accuracy:.2f}%")

Implemented the forward method to flatten input, pass through fc1, apply ReLU, then pass through fc2.

Returned the final output tensor from the forward method.

Results Interpretation

Before Fix: Training loss was high (~2.3), accuracy was very low (~10%), indicating random guessing.

After Fix: Training loss decreased significantly (~0.5), accuracy improved to above 80%, showing the model learned meaningful patterns.

The forward method defines how data flows through the model layers. Correctly implementing it is essential for the model to learn and make accurate predictions.

Bonus Experiment

Add a dropout layer after the ReLU activation in the forward method to reduce overfitting and observe the effect on validation accuracy.

💡 Hint

Use nn.Dropout with a probability like 0.5 and apply it after the ReLU activation in the forward method.

Practice

1. What is the main purpose of the forward method in a PyTorch model?

easy

A. To define how input data passes through the model layers

B. To initialize model weights

C. To save the model to disk

D. To load data from a dataset

Solution

1. Step 1: Understand the role of forward in PyTorch

   The forward method specifies the computation performed on input data as it passes through the model.

2. Step 2: Differentiate from other methods

   Initialization and saving/loading are handled by other methods, not forward.

3. Final Answer:

   To define how input data passes through the model layers -> Option A

4. Quick Check:

   forward defines data flow [OK]

Hint: Remember: forward = data flow through model [OK]

Common Mistakes:

• Confusing forward with initialization
• Thinking forward saves or loads data
• Assuming forward is called manually

2. Which of the following is the correct way to define the forward method inside a PyTorch nn.Module subclass?

easy

A. def Forward(self, x): return x * 2

B. def forward(self, x): return x * 2

C. def forward(x): return x * 2

D. def forward(self): return x * 2

Solution

1. Step 1: Check method signature requirements

   The forward method must have self as the first argument and input data as the second.

2. Step 2: Validate method name and parameters

   def forward(self, x): return x * 2 correctly uses lowercase forward and includes self and x. Others have wrong case, missing self, or missing input.

3. Final Answer:

   def forward(self, x): return x * 2 -> Option B

4. Quick Check:

   Correct method name and parameters = def forward(self, x): return x * 2 [OK]

Hint: Always use lowercase 'forward' with self and input args [OK]

Common Mistakes:

• Capitalizing 'Forward' instead of 'forward'
• Omitting 'self' parameter
• Missing input argument in method

3. Given the following PyTorch model code, what will be the output when calling model(torch.tensor([2.0]))?

import torch
import torch.nn as nn

class SimpleModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(1, 1)
        self.linear.weight.data.fill_(2.0)
        self.linear.bias.data.fill_(1.0)
    def forward(self, x):
        return self.linear(x)

model = SimpleModel()

medium

A. tensor([5.0])

B. tensor([4.0])

C. tensor([2.0])

D. tensor([1.0])

Solution

1. Step 1: Calculate linear layer output

   The linear layer output is weight * input + bias = 2.0 * 2.0 + 1.0 = 5.0.

2. Step 2: Confirm output tensor

   The model returns tensor([5.0]) as output.

3. Final Answer:

   tensor([5.0]) -> Option A

4. Quick Check:

   Output = 2*2 + 1 = 5 [OK]

Hint: Output = weight * input + bias in linear layer [OK]

Common Mistakes:

• Ignoring bias term
• Using wrong weight value
• Confusing input tensor shape

4. Identify the error in this PyTorch model's forward method:

class MyModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc = nn.Linear(3, 1)
    def forward(self, x):
        x = self.fc
        return x

medium

A. Input tensor shape is incorrect

B. Missing call to super().__init__() in __init__

C. The forward method should not have a return statement

D. The forward method returns the layer object, not the output

Solution

1. Step 1: Analyze the forward method code

   The line x = self.fc assigns the layer object to x instead of calling it.

2. Step 2: Identify correct usage

   To get output, the layer must be called with input: x = self.fc(x).

3. Final Answer:

   The forward method returns the layer object, not the output -> Option D

4. Quick Check:

   Layer must be called with input, not assigned [OK]

Hint: Call layers with input inside forward, don't assign layer itself [OK]

Common Mistakes:

• Assigning layer instead of calling it
• Forgetting to call super().__init__()
• Ignoring input tensor shape

5. You want to create a PyTorch model that applies two linear layers sequentially in the forward method, with a ReLU activation in between. Which forward method implementation is correct?

class TwoLayerModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(4, 3)
        self.fc2 = nn.Linear(3, 2)
        self.relu = nn.ReLU()

    def forward(self, x):
        ?

hard

A. x = self.fc1(x)
x = self.fc2(x)
x = self.relu(x)
return x

B. x = self.relu(self.fc1(x))
return self.fc2(x)

C. x = self.fc1(x)
x = self.relu(x)
x = self.fc2(x)
return x

D. x = self.fc1(x)
return self.fc2(self.relu(x))

Solution

1. Step 1: Understand the desired layer order

   The model should apply fc1, then ReLU, then fc2 in sequence.

2. Step 2: Check each option's order

   x = self.fc1(x) x = self.relu(x) x = self.fc2(x) return x correctly applies fc1, then ReLU, then fc2. Others change the order or skip activation between layers.

3. Final Answer:

   x = self.fc1(x) x = self.relu(x) x = self.fc2(x) return x -> Option C

4. Quick Check:

   Layer1 -> ReLU -> Layer2 sequence = x = self.fc1(x) x = self.relu(x) x = self.fc2(x) return x [OK]

Hint: Apply activation between layers, not after last layer [OK]

Common Mistakes:

• Applying activation after last layer
• Skipping activation between layers
• Incorrect order of operations