Why __init__ for layers in PyTorch? - Purpose & Use Cases

The Big Idea

What if you could build complex layers without rewriting the same setup again and again?

The Scenario

Imagine building a complex machine learning model by manually setting up each layer's details every time you want to create it. You have to remember all the settings and write repetitive code for each layer.

The Problem

This manual way is slow and confusing. You might forget important settings or make mistakes that cause your model to fail. It's like assembling a puzzle without a picture to guide you.

The Solution

Using the __init__ method in layer classes lets you set up all the layer's parts once. Then, you can create layers easily and correctly every time by just calling the class. It keeps your code clean and safe.

Before vs After

✗ Before

layer = nn.Linear(10, 5)
layer.weight = torch.randn(5, 10)
layer.bias = torch.zeros(5)

✓ After

class MyLayer(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(10, 5)

What It Enables

It makes building and reusing layers simple, reliable, and fast, so you can focus on designing better models.

Real Life Example

When creating a neural network for image recognition, you can define each layer once with __init__ and then build the whole network quickly without repeating setup code.

Key Takeaways

Manual layer setup is slow and error-prone.

__init__ organizes layer setup in one place.

This leads to cleaner, reusable, and safer model code.

Practice

1. What is the main purpose of the __init__ method in a PyTorch layer class?

easy

A. To save the model to disk

B. To set up the parts of the layer like linear or dropout layers

C. To train the model on data

D. To run the forward pass of the model

Solution

1. Step 1: Understand the role of __init__ in PyTorch

   The __init__ method is used to prepare the layer by defining its parts, such as linear or dropout layers.

2. Step 2: Differentiate from other methods

   The forward pass runs the data through the layer, training updates weights, and saving stores the model. These are not done in __init__.

3. Final Answer:

   To set up the parts of the layer like linear or dropout layers -> Option B

4. Quick Check:

   __init__ prepares layer parts = D [OK]

Hint: Remember: __init__ builds the layer parts first [OK]

Common Mistakes:

• Confusing __init__ with the forward method
• Thinking __init__ trains the model
• Assuming __init__ saves the model

2. Which of the following is the correct way to start the __init__ method in a PyTorch layer class?

easy

A. def __init__(self): super()

B. def __init__(): super().__init__()

C. def __init__(self): super().__init__()

D. def __init__(self): init.super()

Solution

1. Step 1: Check the method signature

   The __init__ method must have self as the first parameter.

2. Step 2: Call the parent class initializer correctly

   Use super().__init__() to properly initialize the parent class.

3. Final Answer:

   def __init__(self): super().__init__() -> Option C

4. Quick Check:

   Correct __init__ start = B [OK]

Hint: Always include self and call super().__init__() first [OK]

Common Mistakes:

• Omitting self parameter
• Calling super() without __init__()
• Wrong method signature without self

3. What will be the output shape of the linear layer defined below inside __init__ when input tensor has shape (10, 5)?

class MyLayer(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = torch.nn.Linear(5, 3)

    def forward(self, x):
        return self.linear(x)

layer = MyLayer()
input_tensor = torch.randn(10, 5)
output = layer(input_tensor)
print(output.shape)

medium

A. torch.Size([10, 3])

B. torch.Size([5, 3])

C. torch.Size([10, 5])

D. torch.Size([3, 10])

Solution

1. Step 1: Understand the linear layer dimensions

   The linear layer is defined as Linear(5, 3), so it expects input features of size 5 and outputs size 3.

2. Step 2: Match input tensor shape and output shape

   The input tensor has shape (10, 5), meaning 10 samples with 5 features each. The output will have shape (10, 3) after the linear layer.

3. Final Answer:

   torch.Size([10, 3]) -> Option A

4. Quick Check:

   Input features 5 to output 3, batch 10 = torch.Size([10, 3]) [OK]

Hint: Output shape = (batch size, output features) [OK]

Common Mistakes:

• Confusing input and output feature sizes
• Mixing batch size with feature dimension
• Ignoring the linear layer's output size

4. Identify the error in the following __init__ method of a PyTorch layer:

class MyLayer(torch.nn.Module):
    def __init__(self):
        self.linear = torch.nn.Linear(4, 2)
        super().__init__()

medium

A. Missing return statement in __init__

B. self.linear should be defined outside __init__

C. No error, code is correct

D. super().__init__() must be called before defining layers

Solution

1. Step 1: Check order of super().__init__() call

   In PyTorch, super().__init__() must be called first inside __init__ to properly initialize the parent class.

2. Step 2: Identify the error in the code

   The code calls super().__init__() after defining self.linear, which is incorrect and can cause errors.

3. Final Answer:

   super().__init__() must be called before defining layers -> Option D

4. Quick Check:

   Call super().__init__() first in __init__ = C [OK]

Hint: Always call super().__init__() before defining layers [OK]

Common Mistakes:

• Defining layers before calling super().__init__()
• Thinking __init__ needs a return statement
• Defining layers outside __init__

5. You want to create a custom PyTorch layer that has two linear layers and a dropout layer. Which __init__ method correctly sets up these parts?

hard

A. def __init__(self):
super().__init__()
self.linear1 = torch.nn.Linear(10, 20)
self.linear2 = torch.nn.Linear(20, 5)
self.dropout = torch.nn.Dropout(0.3)

B. def __init__(self):
self.linear1 = torch.nn.Linear(10, 20)
self.linear2 = torch.nn.Linear(20, 5)
self.dropout = torch.nn.Dropout(0.3)
super().__init__()

C. def __init__(self):
super().__init__()
self.linear1 = torch.nn.Linear(20, 10)
self.linear2 = torch.nn.Linear(5, 20)
self.dropout = torch.nn.Dropout(0.3)

D. def __init__(self):
super().__init__()
self.linear1 = torch.nn.Linear(10, 20)
self.linear2 = torch.nn.Linear(5, 20)
self.dropout = torch.nn.Dropout(0.3)

Solution

1. Step 1: Check the order of super().__init__() call

   It must be called first to initialize the parent class properly.

2. Step 2: Verify layer dimensions and definitions

   Linear layers should connect properly: first from 10 to 20, second from 20 to 5. Dropout is correctly defined with 0.3 probability.

3. Step 3: Identify the correct option

   def __init__(self): super().__init__() self.linear1 = torch.nn.Linear(10, 20) self.linear2 = torch.nn.Linear(20, 5) self.dropout = torch.nn.Dropout(0.3) matches correct order, layer sizes, and dropout setup. Codes that call super().__init__() last or have mismatched dimensions like Linear(20,10) and Linear(5,20) are incorrect.

4. Final Answer:

   def __init__(self): super().__init__() self.linear1 = torch.nn.Linear(10, 20) self.linear2 = torch.nn.Linear(20, 5) self.dropout = torch.nn.Dropout(0.3) -> Option A

5. Quick Check:

   Correct order and layer sizes = A [OK]

Hint: Call super first, then define layers with matching sizes [OK]

Common Mistakes:

• Calling super().__init__() last
• Using wrong input/output sizes for linear layers
• Misplacing dropout layer definition