Broadcasting in PyTorch - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the output shape after broadcasting?

Given two tensors a and b in PyTorch, what is the shape of the result after adding them?

PyTorch

import torch

a = torch.randn(3, 1, 5)
b = torch.randn(1, 4, 1)
result = a + b
output_shape = result.shape
print(output_shape)

A torch.Size([3, 4, 5])

B torch.Size([3, 4, 1])

C torch.Size([1, 4, 1])

D torch.Size([3, 1, 5])

💡 Hint

Remember PyTorch broadcasts dimensions by expanding size 1 dimensions to match the other tensor.

Tensor a has shape (3,1,5) and b has shape (1,4,1). Broadcasting expands the 1s to match the other tensor's size, resulting in (3,4,5).

❓ Model Choice

intermediate

Which operation uses broadcasting correctly?

Which PyTorch operation below correctly uses broadcasting to multiply a tensor of shape (2,3) with a tensor of shape (3,)?

A

x = torch.randn(2,3)
y = torch.randn(2)
result = x * y

B

x = torch.randn(2,3)
y = torch.randn(3)
result = x * y

C

x = torch.randn(2,3)
y = torch.randn(1,3)
result = x + y

D

x = torch.randn(2,3)
y = torch.randn(3,2)
result = x * y

💡 Hint

Broadcasting works when trailing dimensions match or are 1.

Option B multiplies (2,3) by (3,) which broadcasts the (3,) to (2,3). Other options have incompatible shapes for broadcasting.

❓ Hyperparameter

advanced

How does broadcasting affect batch size in model input?

You have a model expecting input shape (batch_size, features). You provide input tensor of shape (features,) without batch dimension. What happens when you add a batch dimension using broadcasting?

A Broadcasting duplicates features along batch dimension, increasing memory usage exponentially.

B The input is broadcasted to shape (batch_size, features) automatically during model forward pass.

C You must manually add batch dimension; broadcasting does not add batch size dimension.

D The model raises an error because broadcasting cannot add missing dimensions.

💡 Hint

Broadcasting can expand size 1 dimensions but cannot add missing dimensions.

Broadcasting can only expand existing dimensions of size 1. It cannot add a new batch dimension if missing. You must reshape or unsqueeze manually.

🔧 Debug

advanced

Why does this broadcasting operation raise an error?

Consider the code below. Why does it raise a runtime error?

PyTorch

import torch
x = torch.randn(4, 3)
y = torch.randn(2, 3)
z = x + y

A Shapes (4,3) and (2,3) are incompatible for broadcasting because the first dimensions differ and neither is 1.

B The tensors must have the same number of elements, which they do not.

C PyTorch does not support broadcasting for addition operations.

D The tensors have different data types causing the addition to fail.

💡 Hint

Check the rules for broadcasting dimensions from left to right.

Broadcasting requires dimensions to be equal or one of them to be 1. Here, 4 and 2 differ and neither is 1, so addition fails.

🧠 Conceptual

expert

What is the effect of broadcasting on memory usage?

When PyTorch broadcasts a tensor during an operation, what happens to the underlying memory usage?

A Broadcasting duplicates the tensor data on the GPU to speed up computation, increasing memory usage.

B Broadcasting compresses the tensor to save memory during operations.

C Broadcasting creates a new large tensor by copying data to match the broadcasted shape, increasing memory usage.

D Broadcasting creates a view that virtually expands the tensor without copying data, so memory usage stays low.

💡 Hint

Think about how broadcasting avoids unnecessary data duplication.

PyTorch uses strides and views to simulate expanded shapes without copying data, so memory usage remains efficient.