__init__ for layers in PyTorch - Model Pipeline Trace

Model Pipeline - __init__ for layers

This pipeline shows how a simple neural network is built using PyTorch layers initialized in the __init__ method. It demonstrates how data flows through the network, how training improves the model, and how predictions are made.

Data Flow - 5 Stages

1Input Data

1000 rows x 10 features→Raw input features for training→1000 rows x 10 features

[[0.5, 1.2, ..., 0.3], [1.1, 0.7, ..., 0.9], ...]

↓

2Layer Initialization (__init__)

N/A→Define layers: Linear(10->5), ReLU, Linear(5->2)→N/A (model structure defined)

self.fc1 = nn.Linear(10, 5); self.relu = nn.ReLU(); self.fc2 = nn.Linear(5, 2)

↓

3Forward Pass

1000 rows x 10 features→Pass input through layers: fc1 -> relu -> fc2→1000 rows x 2 outputs

[[0.1, 0.9], [0.8, 0.2], ...]

↓

4Loss Calculation

1000 rows x 2 outputs→Calculate loss with target labels→Scalar loss value

loss = 0.45

↓

5Backpropagation and Optimization

Model parameters and loss→Update weights to reduce loss→Updated model parameters

Weights in fc1 and fc2 updated

Training Trace - Epoch by Epoch

Loss
0.9 |*       
0.8 | *      
0.7 |  *     
0.6 |   *    
0.5 |    *   
0.4 |     *  
0.3 |      * 
    +---------
     1 2 3 4 5 Epochs

Epoch	Loss ↓	Accuracy ↑	Observation
1	0.85	0.55	Starting training, loss high, accuracy low
2	0.65	0.68	Loss decreased, accuracy improved
3	0.50	0.75	Model learning well
4	0.40	0.82	Loss continues to drop, accuracy rises
5	0.35	0.85	Training converging nicely

Prediction Trace - 5 Layers

Layer 1: Input Layer

Layer 2: First Linear Layer (fc1)

Layer 3: ReLU Activation

Layer 4: Second Linear Layer (fc2)

Layer 5: Softmax (optional for classification)

Model Quiz - 3 Questions

What is the purpose of defining layers in the __init__ method?

A To run the data through the model

B To create the model structure and initialize weights

C To calculate the loss

D To update the weights after training

Key Insight

Initializing layers in the __init__ method sets up the model's structure and parameters before training. This organization helps the model learn by updating these parameters during training, which reduces loss and improves accuracy. Activation functions like ReLU introduce non-linearity, enabling the model to learn complex patterns.

Practice

1. What is the main purpose of the __init__ method in a PyTorch layer class?

easy

A. To save the model to disk

B. To set up the parts of the layer like linear or dropout layers

C. To train the model on data

D. To run the forward pass of the model

Solution

1. Step 1: Understand the role of __init__ in PyTorch

   The __init__ method is used to prepare the layer by defining its parts, such as linear or dropout layers.

2. Step 2: Differentiate from other methods

   The forward pass runs the data through the layer, training updates weights, and saving stores the model. These are not done in __init__.

3. Final Answer:

   To set up the parts of the layer like linear or dropout layers -> Option B

4. Quick Check:

   __init__ prepares layer parts = D [OK]

Hint: Remember: __init__ builds the layer parts first [OK]

Common Mistakes:

• Confusing __init__ with the forward method
• Thinking __init__ trains the model
• Assuming __init__ saves the model

2. Which of the following is the correct way to start the __init__ method in a PyTorch layer class?

easy

A. def __init__(self): super()

B. def __init__(): super().__init__()

C. def __init__(self): super().__init__()

D. def __init__(self): init.super()

Solution

1. Step 1: Check the method signature

   The __init__ method must have self as the first parameter.

2. Step 2: Call the parent class initializer correctly

   Use super().__init__() to properly initialize the parent class.

3. Final Answer:

   def __init__(self): super().__init__() -> Option C

4. Quick Check:

   Correct __init__ start = B [OK]

Hint: Always include self and call super().__init__() first [OK]

Common Mistakes:

• Omitting self parameter
• Calling super() without __init__()
• Wrong method signature without self

3. What will be the output shape of the linear layer defined below inside __init__ when input tensor has shape (10, 5)?

class MyLayer(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = torch.nn.Linear(5, 3)

    def forward(self, x):
        return self.linear(x)

layer = MyLayer()
input_tensor = torch.randn(10, 5)
output = layer(input_tensor)
print(output.shape)

medium

A. torch.Size([10, 3])

B. torch.Size([5, 3])

C. torch.Size([10, 5])

D. torch.Size([3, 10])

Solution

1. Step 1: Understand the linear layer dimensions

   The linear layer is defined as Linear(5, 3), so it expects input features of size 5 and outputs size 3.

2. Step 2: Match input tensor shape and output shape

   The input tensor has shape (10, 5), meaning 10 samples with 5 features each. The output will have shape (10, 3) after the linear layer.

3. Final Answer:

   torch.Size([10, 3]) -> Option A

4. Quick Check:

   Input features 5 to output 3, batch 10 = torch.Size([10, 3]) [OK]

Hint: Output shape = (batch size, output features) [OK]

Common Mistakes:

• Confusing input and output feature sizes
• Mixing batch size with feature dimension
• Ignoring the linear layer's output size

4. Identify the error in the following __init__ method of a PyTorch layer:

class MyLayer(torch.nn.Module):
    def __init__(self):
        self.linear = torch.nn.Linear(4, 2)
        super().__init__()

medium

A. Missing return statement in __init__

B. self.linear should be defined outside __init__

C. No error, code is correct

D. super().__init__() must be called before defining layers

Solution

1. Step 1: Check order of super().__init__() call

   In PyTorch, super().__init__() must be called first inside __init__ to properly initialize the parent class.

2. Step 2: Identify the error in the code

   The code calls super().__init__() after defining self.linear, which is incorrect and can cause errors.

3. Final Answer:

   super().__init__() must be called before defining layers -> Option D

4. Quick Check:

   Call super().__init__() first in __init__ = C [OK]

Hint: Always call super().__init__() before defining layers [OK]

Common Mistakes:

• Defining layers before calling super().__init__()
• Thinking __init__ needs a return statement
• Defining layers outside __init__

5. You want to create a custom PyTorch layer that has two linear layers and a dropout layer. Which __init__ method correctly sets up these parts?

hard

A. def __init__(self):
super().__init__()
self.linear1 = torch.nn.Linear(10, 20)
self.linear2 = torch.nn.Linear(20, 5)
self.dropout = torch.nn.Dropout(0.3)

B. def __init__(self):
self.linear1 = torch.nn.Linear(10, 20)
self.linear2 = torch.nn.Linear(20, 5)
self.dropout = torch.nn.Dropout(0.3)
super().__init__()

C. def __init__(self):
super().__init__()
self.linear1 = torch.nn.Linear(20, 10)
self.linear2 = torch.nn.Linear(5, 20)
self.dropout = torch.nn.Dropout(0.3)

D. def __init__(self):
super().__init__()
self.linear1 = torch.nn.Linear(10, 20)
self.linear2 = torch.nn.Linear(5, 20)
self.dropout = torch.nn.Dropout(0.3)

Solution

1. Step 1: Check the order of super().__init__() call

   It must be called first to initialize the parent class properly.

2. Step 2: Verify layer dimensions and definitions

   Linear layers should connect properly: first from 10 to 20, second from 20 to 5. Dropout is correctly defined with 0.3 probability.

3. Step 3: Identify the correct option

   def __init__(self): super().__init__() self.linear1 = torch.nn.Linear(10, 20) self.linear2 = torch.nn.Linear(20, 5) self.dropout = torch.nn.Dropout(0.3) matches correct order, layer sizes, and dropout setup. Codes that call super().__init__() last or have mismatched dimensions like Linear(20,10) and Linear(5,20) are incorrect.

4. Final Answer:

   def __init__(self): super().__init__() self.linear1 = torch.nn.Linear(10, 20) self.linear2 = torch.nn.Linear(20, 5) self.dropout = torch.nn.Dropout(0.3) -> Option A

5. Quick Check:

   Correct order and layer sizes = A [OK]

Hint: Call super first, then define layers with matching sizes [OK]

Common Mistakes:

• Calling super().__init__() last
• Using wrong input/output sizes for linear layers
• Misplacing dropout layer definition