Why automatic differentiation enables training in PyTorch

Introduction

Automatic differentiation helps computers learn by quickly finding how to change numbers to make predictions better.

When training a neural network to recognize images.

When adjusting a model to predict house prices accurately.

When improving a recommendation system based on user feedback.

When tuning parameters in a machine learning model automatically.

Syntax

PyTorch

loss.backward()
optimizer.step()

loss.backward() calculates gradients automatically.

optimizer.step() updates model parameters using these gradients.

Examples

This calculates gradients of the loss with respect to model parameters.

PyTorch

loss = criterion(output, target)
loss.backward()

Clear old gradients, compute new ones, then update parameters.

PyTorch

optimizer.zero_grad()
loss.backward()
optimizer.step()

Sample Model

This program shows how automatic differentiation finds gradients to train a simple model that learns y = 2x + 1.

PyTorch

import torch
import torch.nn as nn
import torch.optim as optim

# Simple model: y = wx + b
class SimpleModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(1, 1)
    def forward(self, x):
        return self.linear(x)

# Data: x and y (y = 2x + 1)
x = torch.tensor([[1.0], [2.0], [3.0], [4.0]])
y = torch.tensor([[3.0], [5.0], [7.0], [9.0]])

model = SimpleModel()
criterion = nn.MSELoss()
optimizer = optim.SGD(model.parameters(), lr=0.1)

for epoch in range(5):
    optimizer.zero_grad()  # Clear old gradients
    output = model(x)      # Predict
    loss = criterion(output, y)  # Calculate loss
    loss.backward()        # Compute gradients automatically
    optimizer.step()       # Update weights
    print(f"Epoch {epoch+1}, Loss: {loss.item():.4f}")

Important Notes

Automatic differentiation saves time by calculating gradients for all model parameters at once.

Without it, training complex models would be very slow and error-prone.

Summary

Automatic differentiation finds how to change model numbers to reduce errors.

This process is key to training machine learning models efficiently.