Why RNNs handle sequences in PyTorch - Why Metrics Matter

Metrics & Evaluation - Why RNNs handle sequences

Which metric matters for this concept and WHY

For Recurrent Neural Networks (RNNs) handling sequences, the key metrics depend on the task. For sequence classification, accuracy shows how well the model predicts the correct class for the whole sequence. For sequence generation or prediction, loss (like cross-entropy or mean squared error) measures how close the predicted sequence is to the true sequence. These metrics matter because sequences have order and context, so the model must capture dependencies over time. Good metrics show the model understands sequence patterns well.

Confusion matrix or equivalent visualization (ASCII)

Example confusion matrix for sequence classification (2 classes):

          Predicted
          0     1
Actual 0  50    10
       1  5     35

- True Positives (TP) = 35 (class 1 correctly predicted)
- True Negatives (TN) = 50 (class 0 correctly predicted)
- False Positives (FP) = 10 (class 0 predicted as 1)
- False Negatives (FN) = 5  (class 1 predicted as 0)

Total samples = 50 + 10 + 5 + 35 = 100

From this:
- Precision = TP / (TP + FP) = 35 / (35 + 10) = 0.78
- Recall = TP / (TP + FN) = 35 / (35 + 5) = 0.875
- Accuracy = (TP + TN) / Total = (35 + 50) / 100 = 0.85

Precision vs Recall tradeoff with concrete examples

Imagine an RNN used to detect spam messages in a sequence of emails:

• High Precision: The model marks only very sure spam as spam. Few good emails are wrongly marked spam. But it might miss some spam emails (lower recall).
• High Recall: The model catches almost all spam emails, but some good emails might be wrongly marked as spam (lower precision).

For spam filtering, high precision is important to avoid losing good emails. For medical sequence data detecting disease early, high recall is more important to catch all cases, even if some false alarms happen.

What "good" vs "bad" metric values look like for this use case

For RNNs handling sequences:

• Good: Accuracy above 80% on test data, balanced precision and recall above 75%, and low loss showing the model learns sequence patterns well.
• Bad: Accuracy near random chance (e.g., 50% for binary), very low recall or precision (below 50%), or high loss indicating the model fails to capture sequence dependencies.

Good metrics mean the RNN understands order and context in sequences. Bad metrics mean it struggles with remembering or using past information.

Metrics pitfalls (accuracy paradox, data leakage, overfitting indicators)

• Accuracy paradox: If sequences are imbalanced (one class much more common), high accuracy can be misleading. The model might just predict the common class.
• Data leakage: If future sequence data leaks into training, metrics look better but model won't work on real unseen sequences.
• Overfitting: Training accuracy very high but test accuracy low means the RNN memorizes training sequences but fails to generalize.
• Ignoring sequence order: Metrics might look okay if the model ignores order, but it won't perform well on tasks needing sequence context.

Self-check question

Your RNN model for sequence classification has 98% accuracy but only 12% recall on the positive class. Is it good for production? Why not?

Answer: No, it is not good. The very low recall means the model misses most positive cases, which could be critical (like missing fraud or disease). High accuracy is misleading if the data is imbalanced. You need to improve recall to catch more positive sequences.

Key Result

For RNNs handling sequences, balanced precision and recall with good accuracy show the model captures sequence order and context well.

Practice

1. Why are RNNs especially good at handling sequence data like sentences or time series?

easy

A. Because they use convolution to detect patterns

B. Because they keep a memory of previous inputs using a hidden state

C. Because they process all inputs at once without order

D. Because they ignore past inputs to focus on current data

Solution

1. Step 1: Understand RNN memory mechanism

   RNNs keep a hidden state that stores information from previous inputs, acting like memory.

2. Step 2: Relate memory to sequence handling

   This memory lets RNNs understand order and context in sequences like sentences or time series.

3. Final Answer:

   Because they keep a memory of previous inputs using a hidden state -> Option B

4. Quick Check:

   RNN memory = sequence understanding [OK]

Hint: RNNs remember past inputs to handle sequences [OK]

Common Mistakes:

• Thinking RNNs process all inputs at once
• Confusing RNNs with convolutional networks
• Assuming RNNs ignore past data

2. Which of the following is the correct way to initialize a simple RNN layer in PyTorch?

easy

A. rnn = torch.nn.RNN(input_size=10, hidden_size=20, num_layers=1)

B. rnn = torch.nn.RNNLayer(10, 20)

C. rnn = torch.nn.SimpleRNN(10, 20)

D. rnn = torch.nn.RNN(input_size=20, 10)

Solution

1. Step 1: Recall PyTorch RNN syntax

   PyTorch uses torch.nn.RNN with parameters input_size and hidden_size.

2. Step 2: Check options for correct parameter order and names

   rnn = torch.nn.RNN(input_size=10, hidden_size=20, num_layers=1) correctly uses input_size=10 and hidden_size=20 with num_layers=1.

3. Final Answer:

   rnn = torch.nn.RNN(input_size=10, hidden_size=20, num_layers=1) -> Option A

4. Quick Check:

   Correct PyTorch RNN init = rnn = torch.nn.RNN(input_size=10, hidden_size=20, num_layers=1) [OK]

Hint: Use torch.nn.RNN(input_size, hidden_size) to initialize [OK]

Common Mistakes:

• Using non-existent classes like RNNLayer or SimpleRNN
• Swapping input_size and hidden_size
• Missing required parameters

3. Given the following PyTorch code, what is the shape of the output tensor?

import torch
rnn = torch.nn.RNN(input_size=5, hidden_size=3, num_layers=1)
input_seq = torch.randn(4, 2, 5)  # seq_len=4, batch=2, input_size=5
output, hidden = rnn(input_seq)

medium

A. (4, 3, 2)

B. (2, 4, 3)

C. (4, 2, 3)

D. (2, 3, 4)

Solution

1. Step 1: Understand RNN input and output shapes

   Input shape is (seq_len=4, batch=2, input_size=5). Output shape is (seq_len, batch, hidden_size).

2. Step 2: Apply hidden_size to output shape

   Hidden size is 3, so output shape is (4, 2, 3).

3. Final Answer:

   (4, 2, 3) -> Option C

4. Quick Check:

   Output shape = (seq_len, batch, hidden_size) = (4, 2, 3) [OK]

Hint: Output shape = (seq_len, batch, hidden_size) in PyTorch RNN [OK]

Common Mistakes:

• Mixing batch and sequence dimensions
• Confusing hidden_size with input_size
• Assuming output shape swaps batch and seq_len

4. Identify the error in this PyTorch RNN usage:

rnn = torch.nn.RNN(input_size=8, hidden_size=4)
input_seq = torch.randn(5, 3, 10)  # seq_len=5, batch=3, input_size=10
output, hidden = rnn(input_seq)

medium

A. input_seq has wrong input_size dimension

B. RNN missing num_layers parameter

C. Output unpacking is incorrect

D. RNN hidden_size should be larger than input_size

Solution

1. Step 1: Check input_size consistency

   RNN expects input_size=8 but input_seq has last dimension 10, which is incorrect.

2. Step 2: Verify other parameters

   num_layers is optional and defaults to 1, output unpacking is correct, hidden_size can be smaller than input_size.

3. Final Answer:

   input_seq has wrong input_size dimension -> Option A

4. Quick Check:

   Input size mismatch causes error [OK]

Hint: Input tensor last dim must match RNN input_size [OK]

Common Mistakes:

• Assuming num_layers is mandatory
• Thinking hidden_size must be bigger than input_size
• Misunderstanding output unpacking

5. You want to build an RNN model in PyTorch to predict the next word in a sentence. Which approach best uses RNNs' sequence handling ability?

hard

A. Feed the entire sentence as one vector without sequence order to the RNN

B. Ignore the hidden state and predict next word only from the last input word

C. Use a convolutional layer before the RNN to remove sequence order

D. Feed the sentence word by word to the RNN, updating hidden state each step, then predict the next word from the final output

Solution

1. Step 1: Understand RNN sequence processing

   RNNs process inputs step-by-step, keeping hidden state to remember past words.

2. Step 2: Apply this to next word prediction

   Feeding words one by one and using the final output leverages RNN memory to predict the next word.

3. Final Answer:

   Feed the sentence word by word to the RNN, updating hidden state each step, then predict the next word from the final output -> Option D

4. Quick Check:

   Stepwise input + hidden state = best sequence use [OK]

Hint: Feed sequence stepwise, use hidden state for prediction [OK]

Common Mistakes:

• Feeding entire sentence as one vector loses order
• Ignoring hidden state loses sequence memory
• Using convolution to remove sequence order