Broadcasting in PyTorch

Introduction

Broadcasting helps PyTorch do math on tensors of different shapes easily. It saves time and code by automatically expanding smaller tensors to match bigger ones.

Adding a vector to each row of a matrix without writing loops.

Scaling each channel of an image tensor by a different number.

Applying a bias term to every example in a batch during training.

Combining data from different sources with different shapes in a model.

Performing element-wise operations on tensors where one has fewer dimensions.

Syntax

PyTorch

result = tensor1 + tensor2

PyTorch automatically broadcasts tensors if their shapes are compatible.

Broadcasting works from the rightmost dimensions going left.

Examples

Adds vector b to each row of matrix a using broadcasting.

PyTorch

a = torch.tensor([[1, 2, 3], [4, 5, 6]])
b = torch.tensor([10, 20, 30])
c = a + b

Adds vector x to each row of matrix y using broadcasting.

PyTorch

x = torch.tensor([1, 2, 3])
y = torch.tensor([[10], [20], [30]])
z = x + y

Sample Model

This program shows how PyTorch adds a vector to each row of a matrix automatically using broadcasting.

PyTorch

import torch

# Define a 2x3 tensor
matrix = torch.tensor([[1, 2, 3], [4, 5, 6]])

# Define a 1D tensor with 3 elements
vector = torch.tensor([10, 20, 30])

# Add vector to each row of matrix using broadcasting
result = matrix + vector

print("Matrix shape:", matrix.shape)
print("Vector shape:", vector.shape)
print("Result shape:", result.shape)
print("Result tensor:")
print(result)

Important Notes

Broadcasting only works if the dimensions are compatible: either equal or one of them is 1.

If shapes are not compatible, PyTorch will raise an error.

Broadcasting helps avoid writing loops and makes code faster and cleaner.

Summary

Broadcasting lets PyTorch do math on tensors with different shapes by expanding smaller ones.

It works from the rightmost dimension and requires compatible shapes.

Use broadcasting to write simpler and faster tensor operations.