Backward pass (loss.backward) in PyTorch

Introduction

The backward pass calculates how much each part of the model contributed to the error. This helps the model learn by adjusting itself to make better predictions.

When training a neural network to update its weights.

After calculating the loss to find gradients for optimization.

When you want to perform gradient descent to improve model accuracy.

To compute gradients for all parameters before calling the optimizer step.

Syntax

PyTorch

loss.backward()

This function computes gradients of the loss with respect to model parameters.

It must be called after computing the loss and before updating weights.

Examples

Calculate loss and then compute gradients for all model parameters.

PyTorch

loss = criterion(output, target)
loss.backward()

Use a functional loss and then call backward to get gradients.

PyTorch

loss = torch.nn.functional.mse_loss(predictions, labels)
loss.backward()

Sample Model

This code shows a simple linear model learning to fit points. It calculates loss, calls loss.backward() to get gradients, prints them, updates weights, and shows loss improvement.

PyTorch

import torch
import torch.nn as nn
import torch.optim as optim

# Simple linear model
model = nn.Linear(1, 1)

# Sample input and target
x = torch.tensor([[1.0], [2.0], [3.0]])
y = torch.tensor([[2.0], [4.0], [6.0]])

# Mean squared error loss
criterion = nn.MSELoss()

# Optimizer
optimizer = optim.SGD(model.parameters(), lr=0.1)

# Forward pass
output = model(x)

# Compute loss
loss = criterion(output, y)
print(f"Loss before backward: {loss.item():.4f}")

# Backward pass to compute gradients
loss.backward()

# Print gradients of model parameters
for name, param in model.named_parameters():
    print(f"Gradients for {name}: {param.grad}")

# Update weights
optimizer.step()

# Forward pass after update
output2 = model(x)
loss2 = criterion(output2, y)
print(f"Loss after one update: {loss2.item():.4f}")

Important Notes

Always zero gradients before backward pass if running multiple iterations: optimizer.zero_grad().

Calling loss.backward() accumulates gradients, so clear them to avoid mixing.

Summary

Backward pass computes gradients needed to update model weights.

Call loss.backward() after computing loss.

Use gradients with an optimizer to improve model predictions.