Forward pass, loss, backward, step in PyTorch - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of a simple forward pass and loss calculation

What is the value of the loss after running this PyTorch code snippet?

PyTorch

import torch
import torch.nn as nn

model = nn.Linear(2, 1)
inputs = torch.tensor([[1.0, 2.0]])
target = torch.tensor([[1.0]])

output = model(inputs)
criterion = nn.MSELoss()
loss = criterion(output, target)
print(loss.item())

A Raises a RuntimeError due to shape mismatch

B A positive float value representing the mean squared error loss

C A tensor with shape (2, 1)

D 0.0

💡 Hint

The loss is the mean squared error between the model output and the target.

The model produces an output tensor of shape (1,1). The MSELoss compares this output with the target tensor of the same shape and returns a positive scalar loss value.

❓ Model Choice

intermediate

Choosing the correct optimizer step sequence

Which sequence of PyTorch commands correctly performs a training step including forward pass, loss calculation, backward pass, and optimizer step?

A optimizer.zero_grad(); output = model(inputs); loss = criterion(output, target); loss.backward(); optimizer.step()

B output = model(inputs); loss = criterion(output, target); optimizer.step(); loss.backward(); optimizer.zero_grad()

C loss.backward(); optimizer.zero_grad(); output = model(inputs); loss = criterion(output, target); optimizer.step()

D optimizer.step(); optimizer.zero_grad(); output = model(inputs); loss = criterion(output, target); loss.backward()

💡 Hint

Remember to clear gradients before backward pass and update weights after backward.

The correct order is to zero gradients, do forward pass, compute loss, call backward to compute gradients, then call optimizer.step() to update weights.

🔧 Debug

advanced

Identify the error in backward pass usage

What error will this PyTorch code raise when running the backward pass?

PyTorch

import torch
import torch.nn as nn

model = nn.Linear(3, 1)
inputs = torch.tensor([[1.0, 2.0, 3.0]])
target = torch.tensor([[1.0]])

output = model(inputs)
criterion = nn.MSELoss()
loss = criterion(output, target)
loss.backward(retain_graph=True)
loss.backward()

A RuntimeError: Trying to backward through the graph a second time without retain_graph=True.

B No error, code runs successfully.

C TypeError: backward() got an unexpected keyword argument 'retain_graph'.

D ValueError: Target and output shapes do not match.

💡 Hint

Calling backward twice requires special care with retain_graph.

Calling loss.backward() twice without retain_graph=True on the second call causes a RuntimeError because the computation graph is freed after the first backward call.

❓ Hyperparameter

advanced

Effect of learning rate on optimizer step

If you increase the learning rate in the optimizer, what is the most likely effect on the training step?

A The loss function automatically decreases without training.

B The model weights update more slowly, improving training stability.

C The model weights update more aggressively, which may cause training to diverge.

D The backward pass is skipped to speed up training.

💡 Hint

Higher learning rates mean bigger steps in weight updates.

Increasing learning rate causes larger weight updates which can lead to unstable training and divergence if too high.

🧠 Conceptual

expert

Why zero gradients before backward pass?

Why is it necessary to call optimizer.zero_grad() before loss.backward() in a training loop?

A Because zero_grad() initializes model weights to zero before each update.

B Because zero_grad() runs the forward pass to reset outputs.

C Because zero_grad() clears the optimizer state to avoid memory leaks.

D Because gradients accumulate by default, zeroing prevents mixing gradients from multiple batches.

💡 Hint

Think about how gradients behave across multiple backward calls.

PyTorch accumulates gradients on each backward call, so zero_grad() clears old gradients to avoid mixing updates from different batches.