forward method in PyTorch - Model Pipeline Trace

Model Pipeline - forward method

The forward method in PyTorch defines how input data moves through the layers of a neural network to produce output predictions.

Data Flow - 6 Stages

1Input Data

64 rows x 28 x 28 pixels→Batch of grayscale images representing handwritten digits→64 rows x 28 x 28 pixels

Batch of 64 images, each 28x28 pixels with values 0-1

↓

2Flatten Layer

64 rows x 28 x 28 pixels→Convert each 28x28 image into a 784-length vector→64 rows x 784 columns

Image reshaped from 28x28 to 784 features

↓

3Linear Layer 1

64 rows x 784 columns→Matrix multiplication with weights + bias, output 128 features→64 rows x 128 columns

Each image represented as 128 features

↓

4Activation (ReLU)

64 rows x 128 columns→Apply ReLU to add non-linearity (replace negatives with zero)→64 rows x 128 columns

Negative values become zero, positive stay same

↓

5Linear Layer 2

64 rows x 128 columns→Matrix multiplication with weights + bias, output 10 classes→64 rows x 10 columns

Output scores for 10 digit classes

↓

6Output

64 rows x 10 columns→Raw scores (logits) for each class→64 rows x 10 columns

Scores like [2.3, -1.2, 0.5, ..., 1.1]

Training Trace - Epoch by Epoch

Loss
1.2 |*       
0.8 |  *     
0.5 |    *   
0.35|     *  
0.25|      * 
    +--------
     1 2 3 4 5 Epochs

Epoch	Loss ↓	Accuracy ↑	Observation
1	1.2	0.55	Model starts learning, loss high, accuracy moderate
2	0.8	0.72	Loss decreases, accuracy improves
3	0.5	0.83	Model learns important features, better predictions
4	0.35	0.89	Loss continues to drop, accuracy nearing 90%
5	0.25	0.92	Training converges, good accuracy

Prediction Trace - 5 Layers

Layer 1: Input Image

Layer 2: Flatten Layer

Layer 3: Linear Layer 1

Layer 4: ReLU Activation

Layer 5: Linear Layer 2

Model Quiz - 3 Questions

What does the forward method mainly do in a PyTorch model?

A Loads the training data into the model

B Calculates the loss after prediction

C Defines how input data moves through layers to produce output

D Updates the model weights during training

Key Insight

The forward method is the heart of a PyTorch model. It shows step-by-step how raw input data is transformed by layers and activations to produce predictions. Understanding this flow helps grasp how neural networks learn and make decisions.

Practice

1. What is the main purpose of the forward method in a PyTorch model?

easy

A. To define how input data passes through the model layers

B. To initialize model weights

C. To save the model to disk

D. To load data from a dataset

Solution

1. Step 1: Understand the role of forward in PyTorch

   The forward method specifies the computation performed on input data as it passes through the model.

2. Step 2: Differentiate from other methods

   Initialization and saving/loading are handled by other methods, not forward.

3. Final Answer:

   To define how input data passes through the model layers -> Option A

4. Quick Check:

   forward defines data flow [OK]

Hint: Remember: forward = data flow through model [OK]

Common Mistakes:

• Confusing forward with initialization
• Thinking forward saves or loads data
• Assuming forward is called manually

2. Which of the following is the correct way to define the forward method inside a PyTorch nn.Module subclass?

easy

A. def Forward(self, x): return x * 2

B. def forward(self, x): return x * 2

C. def forward(x): return x * 2

D. def forward(self): return x * 2

Solution

1. Step 1: Check method signature requirements

   The forward method must have self as the first argument and input data as the second.

2. Step 2: Validate method name and parameters

   def forward(self, x): return x * 2 correctly uses lowercase forward and includes self and x. Others have wrong case, missing self, or missing input.

3. Final Answer:

   def forward(self, x): return x * 2 -> Option B

4. Quick Check:

   Correct method name and parameters = def forward(self, x): return x * 2 [OK]

Hint: Always use lowercase 'forward' with self and input args [OK]

Common Mistakes:

• Capitalizing 'Forward' instead of 'forward'
• Omitting 'self' parameter
• Missing input argument in method

3. Given the following PyTorch model code, what will be the output when calling model(torch.tensor([2.0]))?

import torch
import torch.nn as nn

class SimpleModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(1, 1)
        self.linear.weight.data.fill_(2.0)
        self.linear.bias.data.fill_(1.0)
    def forward(self, x):
        return self.linear(x)

model = SimpleModel()

medium

A. tensor([5.0])

B. tensor([4.0])

C. tensor([2.0])

D. tensor([1.0])

Solution

1. Step 1: Calculate linear layer output

   The linear layer output is weight * input + bias = 2.0 * 2.0 + 1.0 = 5.0.

2. Step 2: Confirm output tensor

   The model returns tensor([5.0]) as output.

3. Final Answer:

   tensor([5.0]) -> Option A

4. Quick Check:

   Output = 2*2 + 1 = 5 [OK]

Hint: Output = weight * input + bias in linear layer [OK]

Common Mistakes:

• Ignoring bias term
• Using wrong weight value
• Confusing input tensor shape

4. Identify the error in this PyTorch model's forward method:

class MyModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc = nn.Linear(3, 1)
    def forward(self, x):
        x = self.fc
        return x

medium

A. Input tensor shape is incorrect

B. Missing call to super().__init__() in __init__

C. The forward method should not have a return statement

D. The forward method returns the layer object, not the output

Solution

1. Step 1: Analyze the forward method code

   The line x = self.fc assigns the layer object to x instead of calling it.

2. Step 2: Identify correct usage

   To get output, the layer must be called with input: x = self.fc(x).

3. Final Answer:

   The forward method returns the layer object, not the output -> Option D

4. Quick Check:

   Layer must be called with input, not assigned [OK]

Hint: Call layers with input inside forward, don't assign layer itself [OK]

Common Mistakes:

• Assigning layer instead of calling it
• Forgetting to call super().__init__()
• Ignoring input tensor shape

5. You want to create a PyTorch model that applies two linear layers sequentially in the forward method, with a ReLU activation in between. Which forward method implementation is correct?

class TwoLayerModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(4, 3)
        self.fc2 = nn.Linear(3, 2)
        self.relu = nn.ReLU()

    def forward(self, x):
        ?

hard

A. x = self.fc1(x)
x = self.fc2(x)
x = self.relu(x)
return x

B. x = self.relu(self.fc1(x))
return self.fc2(x)

C. x = self.fc1(x)
x = self.relu(x)
x = self.fc2(x)
return x

D. x = self.fc1(x)
return self.fc2(self.relu(x))

Solution

1. Step 1: Understand the desired layer order

   The model should apply fc1, then ReLU, then fc2 in sequence.

2. Step 2: Check each option's order

   x = self.fc1(x) x = self.relu(x) x = self.fc2(x) return x correctly applies fc1, then ReLU, then fc2. Others change the order or skip activation between layers.

3. Final Answer:

   x = self.fc1(x) x = self.relu(x) x = self.fc2(x) return x -> Option C

4. Quick Check:

   Layer1 -> ReLU -> Layer2 sequence = x = self.fc1(x) x = self.relu(x) x = self.fc2(x) return x [OK]

Hint: Apply activation between layers, not after last layer [OK]

Common Mistakes:

• Applying activation after last layer
• Skipping activation between layers
• Incorrect order of operations