Hidden state management in PyTorch - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is a hidden state in recurrent neural networks (RNNs)?

A hidden state is a memory that stores information from previous inputs in a sequence. It helps the RNN remember past data to influence future predictions.

intermediate

Why do we need to manage hidden states carefully during training in PyTorch?

Because hidden states carry information across time steps, improper management can cause errors like backpropagating through the entire history, leading to high memory use and slow training.

intermediate

What does the method detach() do when applied to a hidden state tensor in PyTorch?

detach() stops the hidden state from tracking gradients backward beyond the current step. This prevents backpropagation through the entire sequence history.

beginner

How do you initialize a hidden state for an RNN in PyTorch?

You create a tensor of zeros with the shape (number_of_layers, batch_size, hidden_size) and set requires_grad=False. This tensor is passed as the initial hidden state.

intermediate

What is the difference between hidden state and cell state in LSTM networks?

The hidden state carries output information, while the cell state carries long-term memory. Both work together to help LSTM remember and forget information.

What is the main purpose of the hidden state in an RNN?

A To initialize weights

B To store information from previous inputs

C To compute loss

D To normalize inputs

The hidden state stores information from previous inputs to help the RNN remember context.

In PyTorch, what does calling hidden_state.detach() do?

A Prevents gradients from flowing back beyond this point

B Deletes the hidden state

C Resets the hidden state to zero

D Copies the hidden state to CPU

detach() stops gradient tracking beyond the current step, avoiding backpropagation through the entire history.

How should you initialize the hidden state for an RNN in PyTorch?

A Ones with shape (input_size, hidden_size)

B Random values with shape (batch, input_size)

C A zero tensor with shape (layers, batch, hidden_size)

D A scalar zero

The hidden state must match the RNN layers and batch size, so a zero tensor with shape (layers, batch, hidden_size) is used.

What happens if you do not detach the hidden state during training?

A The hidden state will reset automatically

B The model will not train

C The model will ignore the hidden state

D Backpropagation will go through all previous time steps, increasing memory use

Without detach(), gradients flow through the entire sequence history, causing high memory use and slow training.

In LSTM, what is the role of the cell state compared to the hidden state?

A Cell state carries long-term memory; hidden state carries output

B Cell state is the input; hidden state is the output

C Cell state is always zero; hidden state changes

D They are the same

The cell state stores long-term memory, while the hidden state carries the output at each step.

Explain how hidden states are managed during training of an RNN in PyTorch and why detaching is important.

Think about how gradients flow through time and how to control that.

Describe the difference between hidden state and cell state in LSTM networks and their roles.

Consider how LSTM keeps information over time.