Hidden state management in PyTorch - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the shape of the hidden state after one forward pass?

Consider the following PyTorch code snippet using an LSTM layer. What will be the shape of the hidden state h_n after running the forward pass?

PyTorch

import torch
import torch.nn as nn

lstm = nn.LSTM(input_size=10, hidden_size=20, num_layers=2)
inputs = torch.randn(5, 3, 10)  # seq_len=5, batch=3, input_size=10
output, (h_n, c_n) = lstm(inputs)

# What is h_n.shape?

A (3, 20)

B (5, 3, 20)

C (2, 3, 20)

D (3, 2, 20)

💡 Hint

Remember the hidden state shape is (num_layers, batch_size, hidden_size).

The hidden state h_n of an LSTM has shape (num_layers, batch_size, hidden_size). Here, num_layers=2, batch_size=3, hidden_size=20, so the shape is (2, 3, 20).

🧠 Conceptual

intermediate

Why do we detach hidden states in RNN training?

In training recurrent neural networks, why is it important to detach the hidden state from the computation graph between batches?

A To convert the hidden state to a numpy array

B To increase the size of the hidden state tensor

C To reset the model weights to initial values

D To prevent backpropagation through the entire history and save memory

💡 Hint

Think about how backpropagation works through time in RNNs.

Detaching the hidden state stops gradients from flowing back through all previous batches, which prevents excessive memory use and long backpropagation chains.

🔧 Debug

advanced

Identify the error in hidden state initialization

What is wrong with the following code snippet for initializing hidden states in a GRU model?

PyTorch

import torch
import torch.nn as nn

gru = nn.GRU(input_size=8, hidden_size=16, num_layers=1)
batch_size = 4
h0 = torch.zeros(1, batch_size, 16)  # Intended hidden state
inputs = torch.randn(10, batch_size, 8)
output, hn = gru(inputs, h0)

A h0 should have shape (num_layers, batch_size, hidden_size), not (batch_size, hidden_size)

B h0 should be initialized with ones, not zeros

C inputs should have batch_size as first dimension, not second

D GRU requires hidden state to be on CPU, not default device

💡 Hint

Check the expected shape of the initial hidden state for GRU layers.

The initial hidden state h0 must have shape (num_layers, batch_size, hidden_size). Here, it is missing the num_layers dimension.

❓ Hyperparameter

advanced

Effect of hidden state size on model capacity

Increasing the hidden state size in an RNN model primarily affects which of the following?

A The number of input features processed per time step

B The model's ability to capture more complex patterns by increasing capacity

C The length of input sequences the model can handle

D The batch size used during training

💡 Hint

Think about what hidden state size controls in an RNN.

The hidden state size controls the dimensionality of the internal memory, allowing the model to represent more complex information.

❓ Metrics

expert

Interpreting hidden state outputs for sequence classification

In a sequence classification task using an LSTM, which hidden state output is typically used as the representation for the entire sequence?

A The last hidden state of the top LSTM layer (h_n at last time step)

B The average of all hidden states across all time steps

C The first hidden state of the bottom LSTM layer

D The cell state (c_n) of the first LSTM layer

💡 Hint

Consider which hidden state summarizes the sequence information after processing.

The last hidden state of the top LSTM layer is commonly used as the fixed-size vector representing the entire input sequence for classification.