Linear (fully connected) layers in PyTorch - Model Metrics & Evaluation

Metrics & Evaluation - Linear (fully connected) layers

Which metric matters for Linear (fully connected) layers and WHY

Linear layers are often used in classification or regression tasks. For classification, accuracy, precision, recall, and F1 score are important to understand how well the model predicts classes. For regression, mean squared error (MSE) or mean absolute error (MAE) show how close predictions are to true values. These metrics help us know if the linear layer is learning useful patterns.

Confusion matrix example for classification with a linear layer

      Actual \ Predicted | Positive | Negative
      -------------------|----------|---------
      Positive           |    50    |   10    
      Negative           |    5     |   35    

      Total samples = 50 + 10 + 5 + 35 = 100

      Precision = TP / (TP + FP) = 50 / (50 + 5) = 0.91
      Recall = TP / (TP + FN) = 50 / (50 + 10) = 0.83
      Accuracy = (TP + TN) / Total = (50 + 35) / 100 = 0.85
      F1 Score = 2 * (Precision * Recall) / (Precision + Recall) ≈ 0.87
    

Precision vs Recall tradeoff with linear layers

Imagine a spam email detector using a linear layer. If it marks too many emails as spam (high recall), it might wrongly block good emails (low precision). If it marks very few emails as spam (high precision), it might miss many spam emails (low recall). Depending on what matters more, we adjust the model or threshold.

For example, in medical diagnosis, missing a disease (low recall) is worse than false alarms (low precision). So, recall is prioritized. Linear layers can be tuned to balance this tradeoff.

What good vs bad metric values look like for linear layers

Good metrics: Accuracy above 80%, precision and recall balanced above 0.8, and F1 score close to 0.85 or higher indicate the linear layer is learning well.

Bad metrics: Accuracy near 50% (random guessing), precision or recall below 0.5, or very unbalanced precision and recall (e.g., precision 0.9 but recall 0.2) show the model struggles to learn or is biased.

Common pitfalls when evaluating linear layers

• Accuracy paradox: High accuracy can be misleading if classes are imbalanced. For example, 95% accuracy if 95% of data is one class.
• Data leakage: If test data leaks into training, metrics look unrealistically good.
• Overfitting: Very high training accuracy but low test accuracy means the linear layer memorized training data but won't generalize.
• Ignoring class imbalance: Not using precision, recall, or F1 when classes are uneven can hide poor performance on minority classes.

Self-check question

Your linear model has 98% accuracy but only 12% recall on fraud cases. Is it good for production? Why or why not?

Answer: No, it is not good. The model misses 88% of fraud cases (low recall), which is dangerous. Even though accuracy is high, it mostly predicts non-fraud correctly but fails to catch fraud. For fraud detection, recall is critical.

Key Result

For linear layers in classification, balanced precision and recall with high F1 score indicate good performance; accuracy alone can be misleading.

Practice

1. What does a torch.nn.Linear layer do in a neural network?

easy

A. It connects every input feature to every output feature with weights and optional bias.

B. It applies a convolution operation on the input data.

C. It reduces the input size by pooling.

D. It normalizes the input features to have zero mean and unit variance.

Solution

1. Step 1: Understand the role of Linear layers

   A Linear layer multiplies input features by weights and adds bias to produce output features.

2. Step 2: Compare with other layer types

   Convolution, pooling, and normalization are different operations, not done by Linear layers.

3. Final Answer:

   It connects every input feature to every output feature with weights and optional bias. -> Option A

4. Quick Check:

   Linear layer = weighted sum + bias [OK]

Hint: Linear layers do weighted sums of inputs to outputs [OK]

Common Mistakes:

• Confusing Linear with convolution layers
• Thinking Linear layers do pooling
• Mixing normalization with Linear layers

2. Which of the following is the correct way to create a Linear layer with 10 inputs and 5 outputs in PyTorch?

easy

A. torch.nn.Linear(10, 5)

B. torch.nn.Linear(out_features=10, in_features=5)

C. torch.nn.Linear(in_features=5, out_features=10)

D. torch.nn.Linear(5, 10)

Solution

1. Step 1: Recall Linear layer constructor parameters

   The constructor is torch.nn.Linear(in_features, out_features).

2. Step 2: Match input and output sizes

   We want 10 inputs and 5 outputs, so in_features=10 and out_features=5.

3. Final Answer:

   torch.nn.Linear(10, 5) -> Option A

4. Quick Check:

   Input size first, output size second [OK]

Hint: Remember: Linear(in_features, out_features) [OK]

Common Mistakes:

• Swapping input and output sizes
• Using keyword arguments in wrong order
• Confusing number of features with batch size

3. What is the output shape of the following PyTorch code?

import torch
layer = torch.nn.Linear(4, 3)
input_tensor = torch.randn(2, 4)
output = layer(input_tensor)
print(output.shape)

medium

A. torch.Size([2, 4])

B. torch.Size([4, 3])

C. torch.Size([3, 4])

D. torch.Size([2, 3])

Solution

1. Step 1: Understand input and layer dimensions

   The input tensor has shape (2, 4), meaning batch size 2 and 4 features per sample. The Linear layer maps 4 inputs to 3 outputs.

2. Step 2: Calculate output shape

   The output shape is (batch size, output features) = (2, 3).

3. Final Answer:

   torch.Size([2, 3]) -> Option D

4. Quick Check:

   Output shape = (batch, out_features) [OK]

Hint: Output shape = (batch size, out_features) [OK]

Common Mistakes:

• Confusing input and output feature dimensions
• Mixing batch size with feature size
• Expecting output shape to match input shape

4. Identify the error in this PyTorch code snippet:

import torch
layer = torch.nn.Linear(3, 2)
input_tensor = torch.randn(4, 4)
output = layer(input_tensor)

medium

A. Input tensor has wrong batch size.

B. Linear layer input features do not match input tensor features.

C. Output tensor shape is incorrect.

D. Missing activation function after Linear layer.

Solution

1. Step 1: Check Linear layer input features

   The Linear layer expects input features of size 3.

2. Step 2: Check input tensor shape

   The input tensor shape is (4, 4), so each sample has 4 features, which does not match the expected 3.

3. Final Answer:

   Linear layer input features do not match input tensor features. -> Option B

4. Quick Check:

   Input features must match layer's in_features [OK]

Hint: Input feature size must match Linear layer's in_features [OK]

Common Mistakes:

• Confusing batch size with feature size
• Ignoring feature dimension mismatch
• Thinking activation function is mandatory

5. You want to build a simple neural network with one hidden Linear layer that takes 8 input features, has 4 hidden units, and outputs 2 classes. Which PyTorch code correctly defines this model?

hard

A. import torch.nn as nn
model = nn.Sequential(
nn.Linear(8, 2),
nn.ReLU(),
nn.Linear(4, 2)
)

B. import torch.nn as nn
model = nn.Sequential(
nn.Linear(4, 8),
nn.ReLU(),
nn.Linear(2, 4)
)

C. import torch.nn as nn
model = nn.Sequential(
nn.Linear(8, 4),
nn.ReLU(),
nn.Linear(4, 2)
)

D. import torch.nn as nn
model = nn.Sequential(
nn.Linear(2, 4),
nn.ReLU(),
nn.Linear(8, 4)
)

Solution

1. Step 1: Define input to hidden layer

   The first Linear layer should map 8 input features to 4 hidden units: nn.Linear(8, 4).

2. Step 2: Define hidden to output layer

   The second Linear layer should map 4 hidden units to 2 output classes: nn.Linear(4, 2).

3. Step 3: Check activation function placement

   ReLU activation is correctly placed between the two Linear layers.

4. Final Answer:

   Correct model definition with layers: Linear(8,4), ReLU, Linear(4,2) -> Option C

5. Quick Check:

   Input -> hidden -> output sizes match [OK]

Hint: Match layer sizes: input->hidden->output with activation in between [OK]

Common Mistakes:

• Swapping input and output sizes in layers
• Placing activation after output layer
• Using wrong feature sizes in layers