Forward pass, loss, backward, step in PyTorch - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of a simple forward pass and loss calculation

What is the value of the loss after running this PyTorch code snippet?

PyTorch

import torch
import torch.nn as nn

model = nn.Linear(2, 1)
inputs = torch.tensor([[1.0, 2.0]])
target = torch.tensor([[1.0]])

output = model(inputs)
criterion = nn.MSELoss()
loss = criterion(output, target)
print(loss.item())

A Raises a RuntimeError due to shape mismatch

B A positive float value representing the mean squared error loss

C A tensor with shape (2, 1)

D 0.0

💡 Hint

The loss is the mean squared error between the model output and the target.

The model produces an output tensor of shape (1,1). The MSELoss compares this output with the target tensor of the same shape and returns a positive scalar loss value.

❓ Model Choice

intermediate

Choosing the correct optimizer step sequence

Which sequence of PyTorch commands correctly performs a training step including forward pass, loss calculation, backward pass, and optimizer step?

A optimizer.zero_grad(); output = model(inputs); loss = criterion(output, target); loss.backward(); optimizer.step()

B output = model(inputs); loss = criterion(output, target); optimizer.step(); loss.backward(); optimizer.zero_grad()

C loss.backward(); optimizer.zero_grad(); output = model(inputs); loss = criterion(output, target); optimizer.step()

D optimizer.step(); optimizer.zero_grad(); output = model(inputs); loss = criterion(output, target); loss.backward()

💡 Hint

Remember to clear gradients before backward pass and update weights after backward.

The correct order is to zero gradients, do forward pass, compute loss, call backward to compute gradients, then call optimizer.step() to update weights.

🔧 Debug

advanced

Identify the error in backward pass usage

What error will this PyTorch code raise when running the backward pass?

PyTorch

import torch
import torch.nn as nn

model = nn.Linear(3, 1)
inputs = torch.tensor([[1.0, 2.0, 3.0]])
target = torch.tensor([[1.0]])

output = model(inputs)
criterion = nn.MSELoss()
loss = criterion(output, target)
loss.backward(retain_graph=True)
loss.backward()

A RuntimeError: Trying to backward through the graph a second time without retain_graph=True.

B No error, code runs successfully.

C TypeError: backward() got an unexpected keyword argument 'retain_graph'.

D ValueError: Target and output shapes do not match.

💡 Hint

Calling backward twice requires special care with retain_graph.

Calling loss.backward() twice without retain_graph=True on the second call causes a RuntimeError because the computation graph is freed after the first backward call.

❓ Hyperparameter

advanced

Effect of learning rate on optimizer step

If you increase the learning rate in the optimizer, what is the most likely effect on the training step?

A The loss function automatically decreases without training.

B The model weights update more slowly, improving training stability.

C The model weights update more aggressively, which may cause training to diverge.

D The backward pass is skipped to speed up training.

💡 Hint

Higher learning rates mean bigger steps in weight updates.

Increasing learning rate causes larger weight updates which can lead to unstable training and divergence if too high.

🧠 Conceptual

expert

Why zero gradients before backward pass?

Why is it necessary to call optimizer.zero_grad() before loss.backward() in a training loop?

A Because zero_grad() initializes model weights to zero before each update.

B Because zero_grad() runs the forward pass to reset outputs.

C Because zero_grad() clears the optimizer state to avoid memory leaks.

D Because gradients accumulate by default, zeroing prevents mixing gradients from multiple batches.

💡 Hint

Think about how gradients behave across multiple backward calls.

PyTorch accumulates gradients on each backward call, so zero_grad() clears old gradients to avoid mixing updates from different batches.

Practice

1. What is the main purpose of the forward pass in a PyTorch model?

easy

A. To compute gradients for backpropagation

B. To update the model weights after training

C. To run input data through the model and get predictions

D. To calculate the loss between predictions and targets

Solution

1. Step 1: Understand the forward pass role

   The forward pass takes input data and processes it through the model layers to produce predictions.

2. Step 2: Differentiate from other steps

   Updating weights is done by the optimizer step, loss calculation measures error, and backward computes gradients.

3. Final Answer:

   To run input data through the model and get predictions -> Option C

4. Quick Check:

   Forward pass = predictions [OK]

Hint: Forward pass means getting predictions from inputs [OK]

Common Mistakes:

• Confusing forward pass with weight update
• Mixing loss calculation with forward pass
• Thinking backward computes predictions

2. Which of the following is the correct PyTorch code to perform a backward pass on a loss tensor named loss?

easy

A. loss.step()

B. loss.backward()

C. loss.forward()

D. loss.update()

Solution

1. Step 1: Identify the backward method

   In PyTorch, calling loss.backward() computes gradients for all tensors with requires_grad=True.

2. Step 2: Check other options

   step() is for optimizers, forward() is for models, and update() is not a PyTorch method.

3. Final Answer:

   loss.backward() -> Option B

4. Quick Check:

   Backward pass uses loss.backward() [OK]

Hint: Use loss.backward() to compute gradients [OK]

Common Mistakes:

• Using step() on loss instead of optimizer
• Calling forward() on loss tensor
• Trying to update loss directly

3. Given the following PyTorch code snippet, what will be printed?

import torch
import torch.nn as nn

model = nn.Linear(2, 1)
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)

inputs = torch.tensor([[1.0, 2.0]])
targets = torch.tensor([[1.0]])

outputs = model(inputs)
loss_fn = nn.MSELoss()
loss = loss_fn(outputs, targets)

loss.backward()
optimizer.step()

print(loss.item())

medium

A. A positive float value representing the loss before weight update

B. Zero, because optimizer.step() resets the loss

C. A tensor object, not a float

D. An error because backward() must be called after step()

Solution

1. Step 1: Understand loss calculation and printing

   The loss is computed before backward and step, so loss.item() returns the scalar loss value as a float.

2. Step 2: Confirm optimizer.step() effect

   Calling optimizer.step() updates weights but does not change the already computed loss tensor value.

3. Final Answer:

   A positive float value representing the loss before weight update -> Option A

4. Quick Check:

   loss.item() shows loss before update [OK]

Hint: Loss printed is before optimizer updates weights [OK]

Common Mistakes:

• Thinking loss becomes zero after optimizer.step()
• Expecting loss to be a tensor when printed
• Calling backward() after step() causes error

4. Identify the error in this PyTorch training snippet:

optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
outputs = model(inputs)
loss = loss_fn(outputs, targets)
optimizer.step()
loss.backward()

medium

A. Missing optimizer.zero_grad() before backward

B. Model parameters not passed to optimizer

C. Using wrong loss function

D. Calling optimizer.step() before loss.backward()

Solution

1. Step 1: Check order of backward and step

   Gradients must be computed by loss.backward() before calling optimizer.step() to update weights.

2. Step 2: Analyze given code order

   The code calls optimizer.step() before loss.backward(), so weights update happens without gradients.

3. Final Answer:

   Calling optimizer.step() before loss.backward() -> Option D

4. Quick Check:

   Backward before step is required [OK]

Hint: Always call loss.backward() before optimizer.step() [OK]

Common Mistakes:

• Swapping backward and step order
• Forgetting to zero gradients
• Assuming step() computes gradients

5. You want to train a PyTorch model for one step with input x and target y. Which sequence of commands correctly performs forward pass, loss calculation, backward pass, and weight update?

hard

A. optimizer.zero_grad(); outputs = model(x); loss = loss_fn(outputs, y); loss.backward(); optimizer.step()

B. outputs = model(x); loss = loss_fn(outputs, y); optimizer.step(); loss.backward(); optimizer.zero_grad()

C. loss = loss_fn(x, y); outputs = model(x); optimizer.zero_grad(); loss.backward(); optimizer.step()

D. optimizer.step(); optimizer.zero_grad(); outputs = model(x); loss = loss_fn(outputs, y); loss.backward()

Solution

1. Step 1: Understand correct training step order

   The correct order is: clear gradients, forward pass, compute loss, backward pass, then update weights.

2. Step 2: Check each option's sequence

   optimizer.zero_grad(); outputs = model(x); loss = loss_fn(outputs, y); loss.backward(); optimizer.step() follows the correct order. Others call optimizer.step() too early or compute loss before forward pass.

3. Final Answer:

   optimizer.zero_grad(); outputs = model(x); loss = loss_fn(outputs, y); loss.backward(); optimizer.step() -> Option A

4. Quick Check:

   Zero grad -> forward -> loss -> backward -> step [OK]

Hint: Clear grads, forward, loss, backward, then step [OK]

Common Mistakes:

• Calling optimizer.step() before backward
• Not zeroing gradients before backward
• Computing loss before forward pass