nn.LSTM layer in PyTorch - Model Metrics & Evaluation

Metrics & Evaluation - nn.LSTM layer

Which metric matters for nn.LSTM layer and WHY

The nn.LSTM layer is used for sequence data like text or time series. The main goal is to predict sequences or classify them correctly. So, metrics like accuracy for classification or mean squared error (MSE) for regression matter most. For classification, accuracy tells how many sequences were predicted right. For regression, MSE shows how close predictions are to true values. These metrics help us know if the LSTM learned useful patterns over time steps.

Confusion matrix example for nn.LSTM classification

      Actual \ Predicted | Positive | Negative
      -------------------|----------|---------
      Positive           |    50    |   10
      Negative           |    5     |   35

      Total samples = 50 + 10 + 5 + 35 = 100

      Precision = TP / (TP + FP) = 50 / (50 + 5) = 0.91
      Recall = TP / (TP + FN) = 50 / (50 + 10) = 0.83
      Accuracy = (TP + TN) / Total = (50 + 35) / 100 = 0.85
    

This confusion matrix shows how well the LSTM classified sequences. TP means correct positive predictions, FP means wrong positive predictions, and so on.

Precision vs Recall tradeoff for nn.LSTM

Imagine an LSTM model detecting spam emails (sequence classification). If it has high precision, it means most emails marked as spam really are spam. This avoids annoying users by wrongly blocking good emails.

If it has high recall, it finds almost all spam emails, but might mark some good emails as spam (false alarms).

Depending on what matters more, you tune the LSTM to balance precision and recall. For spam, high precision is often preferred to avoid blocking good mail.

Good vs Bad metric values for nn.LSTM

Good: Accuracy above 85% for classification, precision and recall both above 80%, and low MSE for regression tasks.

Bad: Accuracy near random guess (e.g., 50% for binary), very low recall (missing many true cases), or very high MSE showing poor predictions.

Good metrics mean the LSTM learned useful sequence patterns. Bad metrics mean it failed to capture time dependencies or overfitted.

Common pitfalls in metrics for nn.LSTM

• Accuracy paradox: High accuracy but poor recall if data is imbalanced (e.g., rare events).
• Data leakage: If future time steps leak into training, metrics look unrealistically good.
• Overfitting: Training metrics very good but validation metrics poor, meaning LSTM memorized sequences instead of generalizing.
• Ignoring sequence length: Metrics averaged over sequences of different lengths can be misleading.

Self-check question

Your LSTM model has 98% accuracy but only 12% recall on fraud detection sequences. Is it good for production? Why or why not?

Answer: No, it is not good. The low recall means the model misses most fraud cases, which is dangerous. Even with high accuracy, missing fraud is costly. You should improve recall before using it in production.

Key Result

For nn.LSTM, accuracy and recall are key metrics; high recall is critical in tasks like fraud detection to avoid missing important cases.

Practice

1. What is the primary purpose of the nn.LSTM layer in PyTorch?

easy

A. To process and remember information from sequences over time

B. To perform image classification using convolution

C. To reduce the dimensionality of data using PCA

D. To generate random numbers for initialization

Solution

1. Step 1: Understand the role of LSTM

   LSTM stands for Long Short-Term Memory, a type of recurrent neural network layer designed to handle sequence data and remember information over time.

2. Step 2: Match purpose with options

   Among the options, only processing and remembering sequence information matches the LSTM's purpose.

3. Final Answer:

   To process and remember information from sequences over time -> Option A

4. Quick Check:

   LSTM purpose = sequence memory [OK]

Hint: LSTM = sequence memory layer, not image or random [OK]

Common Mistakes:

• Confusing LSTM with convolutional layers
• Thinking LSTM reduces data dimension like PCA
• Assuming LSTM generates random numbers

2. Which of the following is the correct way to create an LSTM layer in PyTorch with input size 10 and hidden size 20?

easy

A. nn.LSTM(input=10, hidden=20)

B. nn.LSTM(20, 10)

C. nn.LSTM(10, 20)

D. nn.LSTM(hidden_size=10, input_size=20)

Solution

1. Step 1: Recall nn.LSTM constructor parameters

   The first argument is input_size (features per input), the second is hidden_size (features in hidden state).

2. Step 2: Match correct syntax

   nn.LSTM(10, 20) uses nn.LSTM(10, 20) which correctly sets input_size=10 and hidden_size=20.

3. Final Answer:

   nn.LSTM(10, 20) -> Option C

4. Quick Check:

   Constructor order = input_size, hidden_size [OK]

Hint: First arg input size, second hidden size in nn.LSTM() [OK]

Common Mistakes:

• Swapping input_size and hidden_size
• Using wrong keyword arguments
• Confusing parameter names

3. Given the code below, what is the shape of output after running the LSTM?

import torch
import torch.nn as nn
lstm = nn.LSTM(input_size=5, hidden_size=3, num_layers=1)
inputs = torch.randn(4, 2, 5)  # seq_len=4, batch=2, input_size=5
output, (hn, cn) = lstm(inputs)

medium

A. (4, 2, 3)

B. (2, 4, 3)

C. (4, 3, 2)

D. (2, 3, 4)

Solution

1. Step 1: Understand LSTM input and output shapes

   The input shape is (seq_len, batch, input_size). The output shape is (seq_len, batch, hidden_size).

2. Step 2: Apply given dimensions

   Input shape is (4, 2, 5), hidden_size=3, so output shape is (4, 2, 3).

3. Final Answer:

   (4, 2, 3) -> Option A

4. Quick Check:

   Output shape = (seq_len, batch, hidden_size) [OK]

Hint: Output shape matches (seq_len, batch, hidden_size) [OK]

Common Mistakes:

• Mixing batch and sequence dimensions
• Confusing input_size with hidden_size
• Assuming output shape swaps batch and seq_len

4. What is wrong with this code snippet that tries to create an LSTM layer?

import torch.nn as nn
lstm = nn.LSTM(10)

medium

A. The input size must be a tuple, not an integer

B. It misses the hidden_size argument, causing an error

C. LSTM requires a batch size argument at creation

D. The code is correct and runs without error

Solution

1. Step 1: Check nn.LSTM constructor requirements

   nn.LSTM requires at least two positional arguments: input_size and hidden_size.

2. Step 2: Identify missing argument

   The code only provides input_size=10, missing hidden_size, so it will raise a TypeError.

3. Final Answer:

   It misses the hidden_size argument, causing an error -> Option B

4. Quick Check:

   nn.LSTM needs input_size and hidden_size [OK]

Hint: nn.LSTM needs two sizes: input and hidden [OK]

Common Mistakes:

• Thinking batch size is needed at layer creation
• Assuming input_size can be a tuple
• Believing code runs without error

5. You want to build a model that processes sequences of length 6 with 8 features each. You want the LSTM to output a sequence with 12 features per time step. Which of the following LSTM layer initializations is correct to achieve this?

hard

A. nn.LSTM(input_size=12, hidden_size=8)

B. nn.LSTM(input_size=8, hidden_size=6)

C. nn.LSTM(input_size=6, hidden_size=8)

D. nn.LSTM(input_size=8, hidden_size=12)

Solution

1. Step 1: Identify input_size and hidden_size meanings

   input_size is the number of features per time step in the input sequence. hidden_size is the number of features in the output per time step.

2. Step 2: Match given sequence and desired output

   Input sequences have 8 features, so input_size=8. Desired output features per time step is 12, so hidden_size=12.

3. Final Answer:

   nn.LSTM(input_size=8, hidden_size=12) -> Option D

4. Quick Check:

   Input features = 8, output features = 12 [OK]

Hint: Input size = input features, hidden size = output features [OK]

Common Mistakes:

• Confusing sequence length with input_size
• Swapping input_size and hidden_size
• Using sequence length as hidden_size