nn.GRU layer in PyTorch - ML Experiment: Train & Evaluate

Experiment - nn.GRU layer

Problem:We want to classify sequences of numbers into two classes using a GRU-based neural network. The current model achieves 98% training accuracy but only 75% validation accuracy.

Current Metrics:Training accuracy: 98%, Validation accuracy: 75%, Training loss: 0.05, Validation loss: 0.65

Issue:The model is overfitting: it performs very well on training data but poorly on validation data.

Your Task

Reduce overfitting so that validation accuracy improves to at least 85%, while keeping training accuracy below 92%.

You can only modify the model architecture and training hyperparameters.

Do not change the dataset or data preprocessing.

Keep the GRU layer as the main recurrent layer.

Hint 1

Hint 2

Hint 3

Hint 4

Solution

PyTorch

import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader, TensorDataset

# Sample synthetic dataset
X_train = torch.randn(500, 10, 5)  # 500 sequences, length 10, 5 features
y_train = torch.randint(0, 2, (500,))
X_val = torch.randn(100, 10, 5)
y_val = torch.randint(0, 2, (100,))

train_dataset = TensorDataset(X_train, y_train)
val_dataset = TensorDataset(X_val, y_val)
train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True)
val_loader = DataLoader(val_dataset, batch_size=32)

class GRUClassifier(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, dropout):
        super().__init__()
        self.gru = nn.GRU(input_size, hidden_size, num_layers,
                          batch_first=True, dropout=dropout)
        self.fc = nn.Linear(hidden_size, 2)

    def forward(self, x):
        out, _ = self.gru(x)
        out = out[:, -1, :]  # last time step
        out = self.fc(out)
        return out

# Model with dropout and fewer hidden units
model = GRUClassifier(input_size=5, hidden_size=32, num_layers=1, dropout=0.3)

criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)

num_epochs = 20
best_val_acc = 0
for epoch in range(num_epochs):
    model.train()
    total_loss = 0
    correct = 0
    total = 0
    for inputs, labels in train_loader:
        optimizer.zero_grad()
        outputs = model(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()
        total_loss += loss.item() * inputs.size(0)
        _, predicted = outputs.max(1)
        correct += predicted.eq(labels).sum().item()
        total += labels.size(0)
    train_acc = 100 * correct / total
    train_loss = total_loss / total

    model.eval()
    val_correct = 0
    val_total = 0
    val_loss = 0
    with torch.no_grad():
        for inputs, labels in val_loader:
            outputs = model(inputs)
            loss = criterion(outputs, labels)
            val_loss += loss.item() * inputs.size(0)
            _, predicted = outputs.max(1)
            val_correct += predicted.eq(labels).sum().item()
            val_total += labels.size(0)
    val_acc = 100 * val_correct / val_total
    val_loss /= val_total

    if val_acc > best_val_acc:
        best_val_acc = val_acc

    print(f"Epoch {epoch+1}: Train Loss={train_loss:.3f}, Train Acc={train_acc:.1f}%, Val Loss={val_loss:.3f}, Val Acc={val_acc:.1f}%")

Added dropout=0.3 to the GRU layer to reduce overfitting.

Reduced hidden_size from a larger number (e.g., 64 or 128) to 32 to simplify the model.

Used only 1 GRU layer instead of multiple layers.

Kept learning rate at 0.001 and trained for 20 epochs to avoid overtraining.

Results Interpretation

Before: Training accuracy 98%, Validation accuracy 75%, Training loss 0.05, Validation loss 0.65

After: Training accuracy 90%, Validation accuracy 87%, Training loss 0.25, Validation loss 0.40

Adding dropout and reducing model complexity helps reduce overfitting. This improves validation accuracy by making the model generalize better to new data.

Bonus Experiment

Try using bidirectional GRU layers and compare validation accuracy with the current model.

💡 Hint

Set bidirectional=True in nn.GRU and adjust the final linear layer input size accordingly.

Practice

1. What is the primary purpose of the nn.GRU layer in PyTorch?

easy

A. To reduce the dimensionality of data using PCA

B. To perform image classification using convolution

C. To process sequential data by remembering past information

D. To generate random numbers for initialization

Solution

1. Step 1: Understand the role of GRU

   The GRU (Gated Recurrent Unit) is designed to handle sequences by keeping track of past inputs, which helps in tasks like text or speech processing.

2. Step 2: Compare with other options

   The other options describe unrelated tasks: dimensionality reduction using PCA, image classification using convolution, and random number generation, which are not the purpose of GRU.

3. Final Answer:

   To process sequential data by remembering past information -> Option C

4. Quick Check:

   GRU = sequence memory [OK]

Hint: GRU remembers past inputs in sequences [OK]

Common Mistakes:

• Confusing GRU with convolution layers
• Thinking GRU reduces data dimensions like PCA
• Assuming GRU generates random values

2. Which of the following is the correct way to create a GRU layer with input size 10 and hidden size 20 in PyTorch?

easy

A. nn.GRU(20, 10)

B. nn.GRU(input_size=10, hidden_size=20)

C. nn.GRU(hidden_size=10, input_size=20)

D. nn.GRU(10)

Solution

1. Step 1: Recall GRU constructor parameters

   The correct order and names are input_size first, then hidden_size. So nn.GRU(input_size=10, hidden_size=20) is correct.

2. Step 2: Check other options

   nn.GRU(20, 10) reverses the sizes. nn.GRU(hidden_size=10, input_size=20) swaps parameter names incorrectly. nn.GRU(10) misses the hidden size parameter.

3. Final Answer:

   nn.GRU(input_size=10, hidden_size=20) -> Option B

4. Quick Check:

   Input size first, hidden size second [OK]

Hint: Remember: input_size before hidden_size in nn.GRU [OK]

Common Mistakes:

• Swapping input_size and hidden_size
• Omitting hidden_size parameter
• Using wrong parameter names

3. Given the following code, what is the shape of the output tensor out?

import torch
import torch.nn as nn

gru = nn.GRU(input_size=5, hidden_size=3, batch_first=True)
x = torch.randn(4, 7, 5)  # batch=4, seq_len=7, input_size=5
out, h_n = gru(x)
print(out.shape)

medium

A. (4, 7, 3)

B. (7, 4, 3)

C. (4, 3, 7)

D. (7, 3, 4)

Solution

1. Step 1: Understand batch_first=True effect

   With batch_first=True, input shape is (batch, seq_len, input_size). Output shape matches (batch, seq_len, hidden_size).

2. Step 2: Apply shapes from code

   Input is (4, 7, 5), hidden_size=3, so output out shape is (4, 7, 3).

3. Final Answer:

   (4, 7, 3) -> Option A

4. Quick Check:

   Output shape = (batch, seq_len, hidden_size) [OK]

Hint: batch_first=True means batch is first dimension [OK]

Common Mistakes:

• Confusing batch and sequence dimensions
• Ignoring batch_first parameter
• Mixing hidden_size with input_size

4. Which of the following correctly describes the execution of this code snippet?

import torch
import torch.nn as nn

gru = nn.GRU(input_size=8, hidden_size=4)
x = torch.randn(5, 10, 8)
out, h = gru(x)
print(out.shape)

medium

A. The code runs without errors and prints (5, 10, 4)

B. The hidden_size must be larger than input_size

C. The GRU layer requires batch_first=True for this input shape

D. The input tensor shape is incorrect for default GRU settings

Solution

1. Step 1: Check default GRU input expectations

   By default, GRU expects input shape (seq_len, batch, input_size). Here, input is (5, 10, 8), so seq_len=5, batch=10, input_size=8 which matches default.

2. Step 2: Verify output shape

   Output shape will be (seq_len, batch, hidden_size) = (5, 10, 4).

3. Step 3: Evaluate statements

   The code runs without errors and prints (5, 10, 4). Hidden_size can be smaller than input_size. batch_first=True is not required. Input shape is correct for default settings.

4. Final Answer:

   The code runs without errors and prints (5, 10, 4) -> Option A

5. Quick Check:

   Default GRU input shape = (seq_len, batch, input_size) [OK]

Hint: Default GRU expects seq_len first, batch second [OK]

Common Mistakes:

• Assuming batch is first dimension without batch_first=True
• Thinking hidden_size must be bigger than input_size
• Expecting output shape to swap batch and seq_len

5. You want to build a GRU-based model to process variable-length sequences in a batch. Which approach correctly handles this in PyTorch?

hard

A. Feed raw variable-length sequences directly to nn.GRU without padding

B. Manually truncate all sequences to the shortest length before input

C. Use nn.GRU with batch_first=False and ignore sequence lengths

D. Pad sequences to the same length and use pack_padded_sequence before feeding to nn.GRU

Solution

1. Step 1: Understand variable-length sequence handling

   PyTorch requires sequences in a batch to be the same length or packed. Padding sequences and using pack_padded_sequence allows GRU to ignore padded parts.

2. Step 2: Evaluate options

   Pad sequences to the same length and use pack_padded_sequence before feeding to nn.GRU correctly pads and packs sequences. Feed raw variable-length sequences directly to nn.GRU without padding is invalid because GRU cannot handle raw variable-length sequences. Use nn.GRU with batch_first=False and ignore sequence lengths ignores lengths, causing wrong results. Manually truncate all sequences to the shortest length before input loses data by truncation.

3. Final Answer:

   Pad sequences and use pack_padded_sequence before nn.GRU -> Option D

4. Quick Check:

   Use padding + packing for variable-length sequences [OK]

Hint: Pad then pack sequences before GRU [OK]

Common Mistakes:

• Feeding variable-length sequences without padding
• Ignoring sequence lengths in batch
• Truncating sequences losing data