Sequence classification in PyTorch - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is sequence classification in machine learning?

Sequence classification is the task of assigning a label or category to a whole sequence of data points, such as sentences, time series, or DNA sequences.

beginner

Name a common neural network architecture used for sequence classification.

Recurrent Neural Networks (RNNs), especially Long Short-Term Memory (LSTM) networks, are commonly used for sequence classification because they can remember information from previous steps.

intermediate

In PyTorch, which loss function is typically used for multi-class sequence classification?

CrossEntropyLoss is typically used for multi-class sequence classification tasks in PyTorch because it combines softmax and negative log likelihood loss.

beginner

Why do we use padding in sequence classification models?

Padding makes all sequences the same length by adding special tokens, so they can be processed in batches by the model efficiently.

beginner

What does the output of a sequence classification model represent?

The output is usually a set of scores or probabilities for each class, indicating how likely the input sequence belongs to each category.

Which PyTorch layer is commonly used to process sequences in classification tasks?

A nn.BatchNorm2d

B nn.Conv2d

C nn.Linear

D nn.LSTM

nn.LSTM is designed to handle sequential data, making it suitable for sequence classification.

What is the purpose of the CrossEntropyLoss in sequence classification?

A To measure the difference between predicted class probabilities and true labels

B To normalize input sequences

C To pad sequences to equal length

D To reduce overfitting

CrossEntropyLoss calculates how far the predicted probabilities are from the true labels, guiding the model to improve.

Why do we often use padding in sequence classification models?

A To increase the number of classes

B To make all sequences the same length for batch processing

C To reduce the model size

D To speed up training by skipping sequences

Padding ensures sequences have the same length so they can be processed together in batches.

What does the final output layer in a sequence classification model usually do?

A Generates new sequences

B Removes padding tokens

C Outputs class scores or probabilities

D Encodes input sequences

The final layer outputs scores or probabilities for each class to decide the sequence's label.

Which of these is NOT a typical sequence classification application?

A Image classification of cats and dogs

B DNA sequence classification

C Spam detection in emails

D Sentiment analysis of movie reviews

Image classification is not sequence classification; it deals with images, not sequences.

Explain how an LSTM network helps in sequence classification tasks.

Think about how LSTM remembers information from earlier in the sequence.

Describe the role of padding and batching in training sequence classification models.

Why do we need all sequences to be the same length when training?

Practice

1. What is the main goal of sequence classification in PyTorch?

easy

A. To assign a label to the entire input sequence

B. To predict the next item in the sequence

C. To label each item in the sequence separately

D. To generate a new sequence from the input

Solution

1. Step 1: Understand sequence classification

   Sequence classification means giving one label to the whole sequence, not to individual items.

2. Step 2: Compare options

   Only To assign a label to the entire input sequence describes labeling the entire sequence, which matches the goal of sequence classification.

3. Final Answer:

   To assign a label to the entire input sequence -> Option A

4. Quick Check:

   Sequence classification = label whole sequence [OK]

Hint: Sequence classification labels the whole sequence, not parts [OK]

Common Mistakes:

• Confusing sequence classification with sequence labeling
• Thinking it predicts next sequence item
• Assuming it generates new sequences

2. Which PyTorch module is commonly used to process sequences step-by-step for classification?

easy

A. torch.nn.Conv2d

B. torch.nn.Linear

C. torch.nn.RNN

D. torch.nn.BatchNorm1d

Solution

1. Step 1: Identify sequence processing modules

   RNN (Recurrent Neural Network) modules process sequences step-by-step, capturing order.

2. Step 2: Match options to sequence processing

   Only torch.nn.RNN is designed for sequential data; others serve different purposes.

3. Final Answer:

   torch.nn.RNN -> Option C

4. Quick Check:

   RNN processes sequences stepwise [OK]

Hint: RNN modules handle sequences stepwise in PyTorch [OK]

Common Mistakes:

• Choosing Linear which is for fixed-size input
• Selecting Conv2d meant for images
• Picking BatchNorm which normalizes features

3. Given this PyTorch code snippet for sequence classification, what is the shape of the output tensor?

rnn = torch.nn.RNN(input_size=10, hidden_size=20, batch_first=True)
inputs = torch.randn(5, 7, 10)  # batch=5, seq_len=7, features=10
output, hn = rnn(inputs)
final_output = hn.squeeze(0)

medium

A. [5, 20]

B. [5, 7, 20]

C. [7, 20]

D. [5, 10]

Solution

1. Step 1: Understand RNN output shapes

   Output shape is (batch, seq_len, hidden_size) = (5,7,20). hn shape is (num_layers, batch, hidden_size) = (1,5,20).

2. Step 2: Analyze final_output shape

   hn.squeeze(0) removes the first dimension (num_layers), resulting in (5,20).

3. Final Answer:

   [5, 20] -> Option A

4. Quick Check:

   hn.squeeze(0) shape = [batch, hidden_size] = [5, 20] [OK]

Hint: Squeeze removes layer dim; output shape is batch x hidden size [OK]

Common Mistakes:

• Confusing output and hn shapes
• Not squeezing the layer dimension
• Mixing sequence length with batch size

4. Identify the error in this PyTorch sequence classification model code:

class SeqClassifier(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.rnn = torch.nn.RNN(10, 20, batch_first=True)
        self.fc = torch.nn.Linear(10, 2)
    def forward(self, x):
        out, hn = self.rnn(x)
        out = self.fc(hn.squeeze(0))
        return out

medium

A. The forward method should return hn, not out

B. The RNN input size should be 2, not 10

C. The squeeze(0) should be applied to out, not hn

D. The Linear layer input size should be 20, not 10

Solution

1. Step 1: Check Linear layer input size

   The RNN hidden size is 20, so hn has shape (batch, 20). The Linear layer expects input size 10, which is incorrect.

2. Step 2: Correct Linear input size

   Linear layer input size must match hidden size 20 to process hn correctly.

3. Final Answer:

   The Linear layer input size should be 20, not 10 -> Option D

4. Quick Check:

   Linear input size = hidden size = 20 [OK]

Hint: Linear input size must match RNN hidden size [OK]

Common Mistakes:

• Mismatching Linear input size with hidden size
• Applying squeeze to wrong tensor
• Returning wrong tensor from forward

5. You want to classify sequences of varying lengths using an RNN in PyTorch. Which approach correctly handles different sequence lengths during training?

hard

A. Truncate all sequences to the shortest length without padding

B. Pad sequences to the same length and use pack_padded_sequence before RNN

C. Feed sequences directly without padding or packing

D. Use a Linear layer instead of RNN to avoid sequence length issues

Solution

1. Step 1: Understand variable-length sequence handling

   Sequences must be padded to the same length for batch processing, then packed to ignore padding during RNN.

2. Step 2: Evaluate options

   Pad sequences to the same length and use pack_padded_sequence before RNN uses padding plus pack_padded_sequence, the correct PyTorch method to handle varying lengths efficiently.

3. Final Answer:

   Pad sequences to the same length and use pack_padded_sequence before RNN -> Option B

4. Quick Check:

   Use padding + pack_padded_sequence for variable lengths [OK]

Hint: Pad then pack sequences to handle varying lengths in RNN [OK]

Common Mistakes:

• Ignoring padding and feeding raw sequences
• Truncating sequences losing data
• Replacing RNN with Linear layer incorrectly