Broadcasting in PyTorch - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is broadcasting in PyTorch?

Broadcasting is a way PyTorch automatically expands smaller tensors to match the shape of larger tensors during arithmetic operations, without copying data.

intermediate

How does PyTorch decide if two tensors can be broadcast together?

PyTorch compares tensor shapes from the right. Dimensions must be equal or one of them must be 1. If this holds for all dimensions, broadcasting works.

beginner

Example: What is the result shape of adding a tensor of shape (3, 1) to a tensor of shape (1, 4)?

The result shape is (3, 4). The (3, 1) tensor is broadcast along the second dimension, and the (1, 4) tensor is broadcast along the first dimension.

beginner

Why is broadcasting useful in machine learning?

Broadcasting lets you write simple code that works on tensors of different shapes, saving memory and making operations faster without manual reshaping.

beginner

What happens if two tensors have incompatible shapes for broadcasting?

PyTorch raises a runtime error because it cannot automatically expand the tensors to a common shape for the operation.

Which condition must be true for two tensor dimensions to be broadcast together?

A Both are even numbers

B Both are prime numbers

C They are equal or one is 1

D One is double the other

Broadcasting requires dimensions to be equal or one dimension to be 1.

What is the shape of the result when adding tensors of shapes (5, 1, 3) and (1, 4, 3)?

A (5, 4, 3)

B (5, 1, 3)

C (1, 4, 3)

D (5, 4)

Broadcasting expands both tensors to (5, 4, 3) before addition.

What error occurs if tensors cannot be broadcast together?

A ValueError

B SyntaxError

C TypeError

D RuntimeError

PyTorch raises a RuntimeError when broadcasting fails.

Broadcasting helps to avoid which of the following?

A Using loops for element-wise operations

B All of the above

C Copying data unnecessarily

D Manual tensor reshaping

Broadcasting avoids manual reshaping, loops, and unnecessary data copying.

In PyTorch, broadcasting compares tensor shapes starting from which side?

A Right (last dimension)

B Left (first dimension)

C Middle dimension

D Random dimension

Broadcasting compares shapes starting from the rightmost dimension.

Explain broadcasting in PyTorch and why it is useful in simple terms.

Think about how PyTorch handles adding tensors of different shapes without extra work.

Describe the rules PyTorch uses to decide if two tensors can be broadcast together.

Focus on how PyTorch checks each dimension pair.

Practice

1. What does broadcasting in PyTorch allow you to do?

easy

A. Perform operations on tensors of different shapes by expanding smaller ones

B. Convert tensors to numpy arrays automatically

C. Change the data type of tensors during operations

D. Split tensors into smaller chunks for parallel processing

Solution

1. Step 1: Understand broadcasting concept

   Broadcasting allows PyTorch to perform element-wise operations on tensors with different shapes by expanding the smaller tensor to match the larger one.

2. Step 2: Compare options with definition

   Only Perform operations on tensors of different shapes by expanding smaller ones correctly describes this behavior. Other options describe unrelated tensor operations.

3. Final Answer:

   Perform operations on tensors of different shapes by expanding smaller ones -> Option A

4. Quick Check:

   Broadcasting = expanding smaller tensor shapes [OK]

Hint: Broadcasting expands smaller tensor shapes to match larger ones [OK]

Common Mistakes:

• Confusing broadcasting with data type conversion
• Thinking broadcasting splits tensors
• Assuming broadcasting changes tensor storage

2. Which of the following PyTorch code snippets correctly uses broadcasting to add a tensor of shape (3, 1) to a tensor of shape (3, 4)?

easy

A. x = torch.randn(3, 1)
y = torch.randn(3, 4)
z = y + x.T

B. x = torch.randn(3, 1)
y = torch.randn(4, 3)
z = x + y

C. x = torch.randn(1, 3)
y = torch.randn(3, 4)
z = x + y

D. x = torch.randn(3, 1)
y = torch.randn(3, 4)
z = x + y

Solution

1. Step 1: Check tensor shapes for broadcasting compatibility

   Tensor x has shape (3,1) and y has shape (3,4). Broadcasting works from right to left: 1 and 4 are compatible (1 can expand to 4), and 3 and 3 match.

2. Step 2: Verify each option's shapes and operations

   x = torch.randn(3, 1) y = torch.randn(3, 4) z = x + y adds (3,1) + (3,4) correctly. x = torch.randn(3, 1) y = torch.randn(4, 3) z = x + y has incompatible shapes (3,1) + (4,3). x = torch.randn(1, 3) y = torch.randn(3, 4) z = x + y has (1,3) + (3,4) which is incompatible. x = torch.randn(3, 1) y = torch.randn(3, 4) z = y + x.T transposes x to (1,3), making addition incompatible.

3. Final Answer:

   x = torch.randn(3, 1)\ny = torch.randn(3, 4)\nz = x + y -> Option D

4. Quick Check:

   Broadcasting matches trailing dims: 1 and 4 compatible [OK]

Hint: Broadcasting matches shapes from right; 1 can expand to 4 [OK]

Common Mistakes:

• Mixing up tensor shapes and their order
• Assuming transpose always helps broadcasting
• Ignoring dimension order in broadcasting

3. What is the output shape of the following PyTorch code?

import torch
x = torch.randn(2, 3, 1)
y = torch.randn(1, 3, 4)
z = x + y
print(z.shape)

medium

A. torch.Size([2, 3, 4])

B. torch.Size([2, 3, 1])

C. torch.Size([1, 3, 4])

D. RuntimeError due to incompatible shapes

Solution

1. Step 1: Analyze shapes for broadcasting

   x shape is (2,3,1), y shape is (1,3,4). Broadcasting compares dimensions from right: 1 and 4 compatible (1 expands to 4), 3 and 3 match, 2 and 1 compatible (1 expands to 2).

2. Step 2: Determine resulting shape

   Broadcasted shape is (2,3,4) because dimensions expand to the maximum compatible size.

3. Final Answer:

   torch.Size([2, 3, 4]) -> Option A

4. Quick Check:

   Broadcasted shape = max dims per axis [OK]

Hint: Broadcast shape picks max size per dimension from right [OK]

Common Mistakes:

• Ignoring leading dimension broadcasting
• Assuming shapes must be identical
• Thinking 1 cannot expand to larger size

4. Consider this PyTorch code snippet:

import torch
x = torch.randn(4, 1)
y = torch.randn(3, 4)
z = x + y

What is the error and how to fix it?

medium

A. RuntimeError due to shape mismatch; fix by reshaping y to (4,1)

B. RuntimeError due to shape mismatch; fix by reshaping x to (3,1)

C. RuntimeError due to shape mismatch; fix by reshaping x to (1,4)

D. No error; code runs fine

Solution

1. Step 1: Check tensor shapes and broadcasting rules

   x shape is (4,1), y shape is (3,4). Broadcasting compares from right: 1 and 4 compatible (1 expands to 4), 4 and 3 incompatible. So addition fails.

2. Step 2: Fix shapes for broadcasting

   Reshape x to (3,1) so shapes become (3,1) and (3,4). Now 1 and 4 compatible (1 expands to 4), 3 and 3 match. Addition works.

3. Final Answer:

   RuntimeError due to shape mismatch; fix by reshaping x to (3,1) -> Option B

4. Quick Check:

   Broadcast dims must be compatible from right [OK]

Hint: Match trailing dims; reshape smaller tensor to align [OK]

Common Mistakes:

• Trying to reshape y incorrectly
• Ignoring dimension order in broadcasting
• Assuming transpose fixes all shape errors

5. You have a tensor a of shape (5, 1, 3) and a tensor b of shape (1, 4, 1). You want to add them using broadcasting and then compute the mean over the last dimension. Which code snippet correctly does this?

hard

A. c = a + b.T
result = c.mean(dim=2)

B. c = a + b
result = c.mean(dim=1)

C. c = a + b
result = c.mean(dim=2)

D. c = a + b
result = c.mean(dim=0)

Solution

1. Step 1: Verify broadcasting shapes

   a shape: (5,1,3), b shape: (1,4,1). Broadcasting compares dims from right: 3 and 1 compatible (1 expands to 3), 1 and 4 compatible (1 expands to 4), 5 and 1 compatible (1 expands to 5). Result shape is (5,4,3).

2. Step 2: Compute mean over last dimension

   The last dimension is dim=2 (0-based indexing). So mean(dim=2) computes mean over size 3 dimension, resulting in shape (5,4).

3. Final Answer:

   c = a + b\nresult = c.mean(dim=2) -> Option C

4. Quick Check:

   Broadcast shapes then mean over last dim (2) [OK]

Hint: Broadcast shapes, then mean over last dimension (dim=2) [OK]

Common Mistakes:

• Using wrong dimension for mean
• Transposing b unnecessarily
• Confusing dimension indices