Defining a model class in PyTorch - Model Metrics & Evaluation

Metrics & Evaluation - Defining a model class

Which metric matters for Defining a model class and WHY

When you define a model class in PyTorch, the key metric to focus on is the training loss. This tells you how well your model is learning from the data during training. Loss measures the difference between the model's predictions and the actual answers. A lower loss means the model is getting better at making predictions. Once training loss improves, you can check accuracy or other metrics on test data to see if the model generalizes well.

Confusion matrix or equivalent visualization

For classification models defined in PyTorch, a confusion matrix helps visualize how well the model predicts each class. It shows counts of:

• True Positives (TP): Correct positive predictions
• False Positives (FP): Incorrect positive predictions
• True Negatives (TN): Correct negative predictions
• False Negatives (FN): Missed positive cases

Confusion Matrix Example:
          Predicted
          P     N
Actual P  50    10
       N  5     35

Total samples = 50 + 10 + 5 + 35 = 100

This matrix helps calculate precision, recall, and accuracy to evaluate the model defined in your class.

Precision vs Recall tradeoff with concrete examples

When defining a model class, you decide how it learns and predicts. Depending on the task, you might want to optimize for:

• Precision: How many predicted positives are actually correct? Important when false alarms are costly. Example: Email spam filter - you don't want to mark good emails as spam.
• Recall: How many actual positives did the model find? Important when missing positives is risky. Example: Cancer detection - you want to catch as many cases as possible.

Adjusting your model's structure or training can shift this balance. For example, changing the output activation or loss function in your model class affects precision and recall.

What "good" vs "bad" metric values look like for this use case

For a well-defined PyTorch model class:

• Good: Training loss steadily decreases, validation loss decreases or stabilizes, accuracy improves, and precision/recall are balanced based on the task.
• Bad: Training loss stays high or fluctuates wildly, validation loss increases (overfitting), accuracy is low, or precision/recall are very unbalanced (e.g., high precision but very low recall).

Good metrics mean your model class is correctly defined and learning well. Bad metrics suggest bugs in the model code or poor design choices.

Metrics pitfalls

• Accuracy paradox: High accuracy can be misleading if classes are imbalanced. For example, if 95% of data is one class, predicting that class always gives 95% accuracy but no real learning.
• Data leakage: If your model class accidentally uses test data during training, metrics will look unrealistically good.
• Overfitting indicators: Training loss very low but validation loss high means the model memorizes training data but fails on new data.
• Incorrect metric calculation: Using wrong formulas for precision or recall leads to wrong conclusions about model quality.

Self-check question

Your PyTorch model class training shows 98% accuracy but only 12% recall on fraud cases. Is it good for production? Why or why not?

Answer: No, it is not good. The model misses 88% of fraud cases (low recall), which is dangerous because many frauds go undetected. High accuracy is misleading here because fraud cases are rare. You should improve recall to catch more fraud.

Key Result

Training loss guides model learning; precision and recall reveal prediction quality and tradeoffs.

Practice

1. What is the main purpose of defining a model class in PyTorch?

easy

A. To automatically tune hyperparameters

B. To organize layers and define how data flows through the model

C. To store training data separately

D. To visualize the model architecture

Solution

1. Step 1: Understand the role of a model class

   In PyTorch, a model class helps organize the layers and defines the forward pass, which is how data flows through the model.

2. Step 2: Compare with other options

   Storing training data, tuning hyperparameters, or visualizing architecture are not the main purposes of defining the model class.

3. Final Answer:

   To organize layers and define how data flows through the model -> Option B

4. Quick Check:

   Model class = organize layers + data flow [OK]

Hint: Model class defines layers and data flow [OK]

Common Mistakes:

• Thinking model class stores data
• Confusing model class with training process
• Assuming model class tunes hyperparameters

2. Which of the following is the correct way to start defining a PyTorch model class?

easy

A. class MyModel(nn.Module):

B. class MyModel():

C. def MyModel(nn.Module):

D. model = nn.Module()

Solution

1. Step 1: Recall PyTorch model class syntax

   PyTorch models must inherit from nn.Module, so the class definition should be class MyModel(nn.Module):.

2. Step 2: Check other options

   class MyModel(): misses inheritance, C uses def instead of class, and D creates an instance, not a class.

3. Final Answer:

   class MyModel(nn.Module): -> Option A

4. Quick Check:

   Model class inherits nn.Module [OK]

Hint: Model class must inherit nn.Module [OK]

Common Mistakes:

• Forgetting to inherit nn.Module
• Using def instead of class
• Creating instance instead of class

3. Given this PyTorch model class snippet, what will print(model(torch.tensor([[1.0, 2.0]]))) output?

import torch
import torch.nn as nn

class SimpleModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(2, 1)
    def forward(self, x):
        return self.linear(x)

model = SimpleModel()
with torch.no_grad():
    model.linear.weight.fill_(1.0)
    model.linear.bias.fill_(0.0)

medium

A. tensor([[2.]])

B. RuntimeError due to wrong input shape

C. tensor([[0.]])

D. tensor([[3.]])

Solution

1. Step 1: Understand the model and weights

   The model has a linear layer with 2 inputs and 1 output. We set weights to 1.0 and bias to 0.0.

2. Step 2: Calculate output for input tensor

   Input is [[1.0, 2.0]]. The output is (1*1.0 + 2*1.0) + 0 = 3.0, so output tensor is [[3.0]].

3. Final Answer:

   tensor([[3.]]) -> Option D

4. Quick Check:

   Sum inputs * weights + bias = 3.0 [OK]

Hint: Sum inputs times weights plus bias for linear output [OK]

Common Mistakes:

• Ignoring bias value
• Mixing input shape
• Assuming default weights instead of set values

4. What is wrong with this PyTorch model class definition?

import torch.nn as nn

class BadModel(nn.Module):
    def __init__(self):
        self.linear = nn.Linear(3, 1)
    def forward(self, x):
        return self.linear(x)

medium

A. Class should not inherit from nn.Module

B. forward method should be named backward

C. Missing call to super().__init__() in __init__ method

D. nn.Linear should have input size 1, not 3

Solution

1. Step 1: Check __init__ method

   In PyTorch, the __init__ method must call super().__init__() to initialize the base nn.Module class.

2. Step 2: Verify other parts

   The forward method name is correct, input size 3 is valid, and inheriting nn.Module is required.

3. Final Answer:

   Missing call to super().__init__() in __init__ method -> Option C

4. Quick Check:

   Always call super().__init__() in __init__ [OK]

Hint: Always call super().__init__() in model __init__ [OK]

Common Mistakes:

• Forgetting super().__init__()
• Renaming forward method incorrectly
• Changing input size without reason

5. You want to define a PyTorch model class with two linear layers and a ReLU activation between them. Which code correctly implements this?

hard

A. class TwoLayerModel(nn.Module):
def __init__(self):
super().__init__()
self.layer1 = nn.Linear(4, 3)
self.layer2 = nn.Linear(3, 1)
self.relu = nn.ReLU()
def forward(self, x):
x = self.layer1(x)
x = self.relu(x)
x = self.layer2(x)
return x

B. class TwoLayerModel(nn.Module):
def __init__(self):
self.layer1 = nn.Linear(4, 3)
self.layer2 = nn.Linear(3, 1)
self.relu = nn.ReLU()
def forward(self, x):
x = self.layer1(x)
x = self.relu(x)
x = self.layer2(x)
return x

C. class TwoLayerModel(nn.Module):
def __init__(self):
super().__init__()
self.layer1 = nn.Linear(3, 4)
self.layer2 = nn.Linear(1, 3)
self.relu = nn.ReLU()
def forward(self, x):
x = self.layer1(x)
x = self.relu(x)
x = self.layer2(x)
return x

D. class TwoLayerModel:
def __init__(self):
super().__init__()
self.layer1 = nn.Linear(4, 3)
self.layer2 = nn.Linear(3, 1)
self.relu = nn.ReLU()
def forward(self, x):
x = self.layer1(x)
x = self.relu(x)
x = self.layer2(x)
return x

Solution

1. Step 1: Check inheritance and super call

   class TwoLayerModel(nn.Module): def __init__(self): super().__init__() self.layer1 = nn.Linear(4, 3) self.layer2 = nn.Linear(3, 1) self.relu = nn.ReLU() def forward(self, x): x = self.layer1(x) x = self.relu(x) x = self.layer2(x) return x correctly inherits nn.Module and calls super().__init__(). class TwoLayerModel(nn.Module): def __init__(self): self.layer1 = nn.Linear(4, 3) self.layer2 = nn.Linear(3, 1) self.relu = nn.ReLU() def forward(self, x): x = self.layer1(x) x = self.relu(x) x = self.layer2(x) return x misses super().__init__(), and D does not inherit nn.Module.

2. Step 2: Verify layer sizes and forward method

   class TwoLayerModel(nn.Module): def __init__(self): super().__init__() self.layer1 = nn.Linear(4, 3) self.layer2 = nn.Linear(3, 1) self.relu = nn.ReLU() def forward(self, x): x = self.layer1(x) x = self.relu(x) x = self.layer2(x) return x uses correct input-output sizes (4->3, 3->1) and applies ReLU between layers. class TwoLayerModel(nn.Module): def __init__(self): super().__init__() self.layer1 = nn.Linear(3, 4) self.layer2 = nn.Linear(1, 3) self.relu = nn.ReLU() def forward(self, x): x = self.layer1(x) x = self.relu(x) x = self.layer2(x) return x has wrong layer sizes.

3. Final Answer:

   Option A correctly defines the model with two linear layers and ReLU -> Option A

4. Quick Check:

   Inherit nn.Module + super().__init__() + correct layers [OK]

Hint: Inherit nn.Module, call super().__init__(), correct layer sizes [OK]

Common Mistakes:

• Forgetting super().__init__()
• Wrong input-output sizes in layers
• Not inheriting nn.Module