Training loop structure in PyTorch - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the output of this PyTorch training loop snippet?

Consider the following PyTorch training loop snippet. What will be the printed loss value after one batch?

PyTorch

import torch
import torch.nn as nn

model = nn.Linear(2, 1)
criterion = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)

inputs = torch.tensor([[1.0, 2.0]])
targets = torch.tensor([[1.0]])

optimizer.zero_grad()
outputs = model(inputs)
loss = criterion(outputs, targets)
loss.backward()
optimizer.step()

print(round(loss.item(), 4))

A 0.0

B 0.5

C 0.25

D 0.75

💡 Hint

Remember the initial weights of nn.Linear are random, so the loss won't be zero.

The initial weights are random, so the output will not match the target exactly. The MSE loss will be a positive value around 0.5 for this input and target.

❓ Model Choice

intermediate

Which part of the training loop updates the model's weights?

In a typical PyTorch training loop, which line is responsible for updating the model's weights?

A optimizer.zero_grad()

B loss.backward()

C optimizer.step()

D criterion(outputs, targets)

💡 Hint

Think about which function applies the calculated gradients to the weights.

optimizer.step() applies the gradients computed by loss.backward() to update the model's weights.

❓ Hyperparameter

advanced

What effect does increasing the learning rate have in a training loop?

If you increase the learning rate in your training loop, what is the most likely effect on training?

A Training may become unstable and loss might not decrease

B Training will always be faster and more stable

C Model will automatically generalize better

D Loss function will change to a different type

💡 Hint

Think about how too large steps affect finding the minimum loss.

A high learning rate can cause the model to overshoot minima, making training unstable and preventing loss from decreasing properly.

🔧 Debug

advanced

Why does this training loop raise an error?

Examine this PyTorch training loop snippet. Why does it raise a RuntimeError?

PyTorch

import torch
import torch.nn as nn

model = nn.Linear(2, 1)
criterion = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)

inputs = torch.tensor([[1.0, 2.0]])
targets = torch.tensor([[1.0]])

outputs = model(inputs)
loss = criterion(outputs, targets)
loss.backward()
optimizer.step()
optimizer.step()  # Called twice without zero_grad

A Calling optimizer.step() twice without zero_grad causes error due to stale gradients

B loss.backward() is missing before optimizer.step()

C optimizer.zero_grad() is called after optimizer.step()

D inputs and targets have mismatched shapes

💡 Hint

Check how many times optimizer.step() is called without clearing gradients.

Calling optimizer.step() twice without clearing gradients accumulates gradients incorrectly, causing a RuntimeError.

🧠 Conceptual

expert

Why do we call optimizer.zero_grad() at the start of each training iteration?

In PyTorch training loops, why is optimizer.zero_grad() called before computing the loss and calling loss.backward()?

A To zero out the loss value before backpropagation

B To initialize model weights to zero

C To clear the optimizer’s learning rate cache

D To reset gradients so they don’t accumulate across batches

💡 Hint

Think about what happens if gradients from previous batches are not cleared.

optimizer.zero_grad() clears old gradients; otherwise, gradients accumulate and cause incorrect updates.