nn.RNN layer in PyTorch - Model Metrics & Evaluation

Metrics & Evaluation - nn.RNN layer

Which metric matters for nn.RNN layer and WHY

The nn.RNN layer is used for sequence data, often in tasks like text or time series prediction. The key metrics depend on the task:

• For classification tasks: Accuracy, Precision, Recall, and F1-score matter to understand how well the RNN predicts correct classes over sequences.
• For regression tasks: Mean Squared Error (MSE) or Mean Absolute Error (MAE) show how close predictions are to true values.

Because RNNs handle sequences, it is important to evaluate metrics that reflect performance over the entire sequence, not just single points.

Confusion matrix example for nn.RNN classification

    Actual \ Predicted | Positive | Negative
    -------------------|----------|---------
    Positive           |    50    |   10    
    Negative           |    5     |   35    

    Total samples = 50 + 10 + 5 + 35 = 100
    

From this matrix:

• True Positives (TP) = 50
• False Positives (FP) = 5
• True Negatives (TN) = 35
• False Negatives (FN) = 10

Precision = TP / (TP + FP) = 50 / (50 + 5) = 0.91

Recall = TP / (TP + FN) = 50 / (50 + 10) = 0.83

F1-score = 2 * (Precision * Recall) / (Precision + Recall) ≈ 0.87

Precision vs Recall tradeoff with nn.RNN

Imagine an RNN used to detect spam messages:

• High Precision: Most messages marked as spam really are spam. This avoids blocking good messages.
• High Recall: Most spam messages are caught, but some good messages might be wrongly blocked.

If the RNN is tuned for high precision, it misses some spam (low recall). If tuned for high recall, it may mark good messages as spam (low precision).

Choosing the right balance depends on what is worse: missing spam or blocking good messages.

Good vs Bad metric values for nn.RNN layer

For classification tasks:

• Good: Precision and Recall above 0.8, F1-score above 0.8, accuracy close to or above 85%.
• Bad: Precision or Recall below 0.5, F1-score below 0.6, accuracy near random chance (e.g., 50% for binary).

For regression tasks:

• Good: Low MSE or MAE, showing predictions close to true values.
• Bad: High error values, indicating poor prediction quality.

Common pitfalls when evaluating nn.RNN layer

• Accuracy paradox: High accuracy can be misleading if classes are imbalanced. For example, if 90% of sequences belong to one class, predicting that class always gives 90% accuracy but poor real performance.
• Data leakage: Using future sequence data in training or validation can inflate metrics falsely.
• Overfitting: Very high training accuracy but low validation accuracy means the RNN memorized training sequences but cannot generalize.
• Ignoring sequence length: Metrics should consider performance across entire sequences, not just individual time steps.

Self-check question

Your nn.RNN model has 98% accuracy but only 12% recall on the fraud class. Is it good for production? Why or why not?

Answer: No, it is not good. The low recall means the model misses 88% of fraud cases, which is dangerous. High accuracy is misleading because fraud cases are rare. The model needs better recall to catch fraud effectively.

Key Result

For nn.RNN layers, precision, recall, and F1-score are key metrics to evaluate sequence classification quality, while error metrics matter for regression.

Practice

1. What does the nn.RNN layer in PyTorch primarily do?

easy

A. Processes sequences step by step, keeping track of past information

B. Sorts input data in ascending order

C. Generates random numbers for initialization

D. Performs matrix multiplication without memory

Solution

1. Step 1: Understand the purpose of RNN

   The RNN layer is designed to handle sequential data by processing one step at a time and remembering previous steps.

2. Step 2: Compare options with RNN behavior

   Only Processes sequences step by step, keeping track of past information describes this behavior correctly; others describe unrelated functions.

3. Final Answer:

   Processes sequences step by step, keeping track of past information -> Option A

4. Quick Check:

   RNN remembers past inputs = A [OK]

Hint: RNNs remember past steps in sequences [OK]

Common Mistakes:

• Thinking RNN sorts data
• Confusing RNN with random number generators
• Assuming RNN does simple matrix multiplication only

2. Which of the following is the correct way to create an RNN layer with input size 10 and hidden size 20 in PyTorch?

easy

A. nn.RNN(20, 10)

B. nn.RNN(10)

C. nn.RNN(input_size=10, hidden_size=20)

D. nn.RNN(hidden_size=10, input_size=20)

Solution

1. Step 1: Recall nn.RNN constructor parameters

   The constructor requires input_size first, then hidden_size, e.g., nn.RNN(input_size=10, hidden_size=20).

2. Step 2: Check each option

   Only nn.RNN(input_size=10, hidden_size=20) matches the correct parameter order and names; the others reverse sizes, omit hidden_size, or swap parameters.

3. Final Answer:

   nn.RNN(input_size=10, hidden_size=20) -> Option C

4. Quick Check:

   Input size first, hidden size second = D [OK]

Hint: Remember: input_size before hidden_size in nn.RNN [OK]

Common Mistakes:

• Swapping input_size and hidden_size
• Omitting hidden_size parameter
• Using positional args in wrong order

3. Given the code below, what is the shape of output after running the RNN?

import torch
import torch.nn as nn
rnn = nn.RNN(input_size=5, hidden_size=3, batch_first=True)
input = torch.randn(4, 7, 5)  # batch=4, seq_len=7, input_size=5
output, hn = rnn(input)

medium

A. (7, 4, 3)

B. (3, 4, 7)

C. (4, 3, 7)

D. (4, 7, 3)

Solution

1. Step 1: Understand batch_first=True effect

   With batch_first=True, input shape is (batch, seq_len, input_size), so output shape is (batch, seq_len, hidden_size).

2. Step 2: Apply shapes to given input

   Input shape is (4, 7, 5), so output shape is (4, 7, 3) because hidden_size=3.

3. Final Answer:

   (4, 7, 3) -> Option D

4. Quick Check:

   Output shape = (batch, seq_len, hidden_size) = B [OK]

Hint: batch_first=True means batch is first dimension [OK]

Common Mistakes:

• Confusing batch and sequence length order
• Ignoring batch_first parameter
• Mixing hidden_size with input_size in output shape

4. What is wrong with this code snippet using nn.RNN?

rnn = nn.RNN(input_size=8, hidden_size=4)
input = torch.randn(3, 5, 10)  # batch=3, seq_len=5, input_size=10
output, hn = rnn(input)

medium

A. RNN requires input to be 2D tensor

B. Input size does not match the RNN's input_size parameter

C. Batch size should be last dimension

D. Hidden size must be equal to input size

Solution

1. Step 1: Check input_size parameter vs input tensor

   The RNN expects input_size=8, but input tensor's last dimension is 10, causing mismatch.

2. Step 2: Validate tensor shape requirements

   Input shape (3, 5, 10) means batch=3, seq_len=5, input_size=10, which conflicts with RNN's input_size=8.

3. Final Answer:

   Input size does not match the RNN's input_size parameter -> Option B

4. Quick Check:

   Input last dim must match input_size = C [OK]

Hint: Input last dimension must match RNN input_size [OK]

Common Mistakes:

• Ignoring input_size mismatch
• Thinking batch size is last dimension
• Assuming RNN input is 2D tensor

5. You want to process a batch of sequences with varying lengths using nn.RNN. Which approach correctly handles this in PyTorch?

hard

A. Pad sequences to the same length and use pack_padded_sequence before the RNN

B. Feed sequences directly without padding or packing

C. Use a for loop to process each sequence separately without padding

D. Set hidden_size equal to the longest sequence length

Solution

1. Step 1: Understand handling variable-length sequences

   PyTorch recommends padding sequences to equal length and using pack_padded_sequence to inform RNN about actual lengths.

2. Step 2: Evaluate options for best practice

   Pad sequences to the same length and use pack_padded_sequence before the RNN correctly describes this approach. Options B and C ignore padding/packing, causing errors or inefficiency. Set hidden_size equal to the longest sequence length is unrelated to sequence length handling.

3. Final Answer:

   Pad sequences to the same length and use pack_padded_sequence before the RNN -> Option A

4. Quick Check:

   Use padding + pack_padded_sequence for variable lengths = A [OK]

Hint: Pad and pack sequences before RNN for variable lengths [OK]

Common Mistakes:

• Feeding raw variable-length sequences directly
• Ignoring packing after padding
• Misusing hidden_size for sequence length