Broadcasting in PyTorch - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the output shape after broadcasting?

Given two tensors a and b in PyTorch, what is the shape of the result after adding them?

PyTorch

import torch

a = torch.randn(3, 1, 5)
b = torch.randn(1, 4, 1)
result = a + b
output_shape = result.shape
print(output_shape)

A torch.Size([3, 4, 5])

B torch.Size([3, 4, 1])

C torch.Size([1, 4, 1])

D torch.Size([3, 1, 5])

💡 Hint

Remember PyTorch broadcasts dimensions by expanding size 1 dimensions to match the other tensor.

Tensor a has shape (3,1,5) and b has shape (1,4,1). Broadcasting expands the 1s to match the other tensor's size, resulting in (3,4,5).

❓ Model Choice

intermediate

Which operation uses broadcasting correctly?

Which PyTorch operation below correctly uses broadcasting to multiply a tensor of shape (2,3) with a tensor of shape (3,)?

A

x = torch.randn(2,3)
y = torch.randn(2)
result = x * y

B

x = torch.randn(2,3)
y = torch.randn(3)
result = x * y

C

x = torch.randn(2,3)
y = torch.randn(1,3)
result = x + y

D

x = torch.randn(2,3)
y = torch.randn(3,2)
result = x * y

💡 Hint

Broadcasting works when trailing dimensions match or are 1.

Option B multiplies (2,3) by (3,) which broadcasts the (3,) to (2,3). Other options have incompatible shapes for broadcasting.

❓ Hyperparameter

advanced

How does broadcasting affect batch size in model input?

You have a model expecting input shape (batch_size, features). You provide input tensor of shape (features,) without batch dimension. What happens when you add a batch dimension using broadcasting?

A Broadcasting duplicates features along batch dimension, increasing memory usage exponentially.

B The input is broadcasted to shape (batch_size, features) automatically during model forward pass.

C You must manually add batch dimension; broadcasting does not add batch size dimension.

D The model raises an error because broadcasting cannot add missing dimensions.

💡 Hint

Broadcasting can expand size 1 dimensions but cannot add missing dimensions.

Broadcasting can only expand existing dimensions of size 1. It cannot add a new batch dimension if missing. You must reshape or unsqueeze manually.

🔧 Debug

advanced

Why does this broadcasting operation raise an error?

Consider the code below. Why does it raise a runtime error?

PyTorch

import torch
x = torch.randn(4, 3)
y = torch.randn(2, 3)
z = x + y

A Shapes (4,3) and (2,3) are incompatible for broadcasting because the first dimensions differ and neither is 1.

B The tensors must have the same number of elements, which they do not.

C PyTorch does not support broadcasting for addition operations.

D The tensors have different data types causing the addition to fail.

💡 Hint

Check the rules for broadcasting dimensions from left to right.

Broadcasting requires dimensions to be equal or one of them to be 1. Here, 4 and 2 differ and neither is 1, so addition fails.

🧠 Conceptual

expert

What is the effect of broadcasting on memory usage?

When PyTorch broadcasts a tensor during an operation, what happens to the underlying memory usage?

A Broadcasting duplicates the tensor data on the GPU to speed up computation, increasing memory usage.

B Broadcasting compresses the tensor to save memory during operations.

C Broadcasting creates a new large tensor by copying data to match the broadcasted shape, increasing memory usage.

D Broadcasting creates a view that virtually expands the tensor without copying data, so memory usage stays low.

💡 Hint

Think about how broadcasting avoids unnecessary data duplication.

PyTorch uses strides and views to simulate expanded shapes without copying data, so memory usage remains efficient.

Practice

1. What does broadcasting in PyTorch allow you to do?

easy

A. Perform operations on tensors of different shapes by expanding smaller ones

B. Convert tensors to numpy arrays automatically

C. Change the data type of tensors during operations

D. Split tensors into smaller chunks for parallel processing

Solution

1. Step 1: Understand broadcasting concept

   Broadcasting allows PyTorch to perform element-wise operations on tensors with different shapes by expanding the smaller tensor to match the larger one.

2. Step 2: Compare options with definition

   Only Perform operations on tensors of different shapes by expanding smaller ones correctly describes this behavior. Other options describe unrelated tensor operations.

3. Final Answer:

   Perform operations on tensors of different shapes by expanding smaller ones -> Option A

4. Quick Check:

   Broadcasting = expanding smaller tensor shapes [OK]

Hint: Broadcasting expands smaller tensor shapes to match larger ones [OK]

Common Mistakes:

• Confusing broadcasting with data type conversion
• Thinking broadcasting splits tensors
• Assuming broadcasting changes tensor storage

2. Which of the following PyTorch code snippets correctly uses broadcasting to add a tensor of shape (3, 1) to a tensor of shape (3, 4)?

easy

A. x = torch.randn(3, 1)
y = torch.randn(3, 4)
z = y + x.T

B. x = torch.randn(3, 1)
y = torch.randn(4, 3)
z = x + y

C. x = torch.randn(1, 3)
y = torch.randn(3, 4)
z = x + y

D. x = torch.randn(3, 1)
y = torch.randn(3, 4)
z = x + y

Solution

1. Step 1: Check tensor shapes for broadcasting compatibility

   Tensor x has shape (3,1) and y has shape (3,4). Broadcasting works from right to left: 1 and 4 are compatible (1 can expand to 4), and 3 and 3 match.

2. Step 2: Verify each option's shapes and operations

   x = torch.randn(3, 1) y = torch.randn(3, 4) z = x + y adds (3,1) + (3,4) correctly. x = torch.randn(3, 1) y = torch.randn(4, 3) z = x + y has incompatible shapes (3,1) + (4,3). x = torch.randn(1, 3) y = torch.randn(3, 4) z = x + y has (1,3) + (3,4) which is incompatible. x = torch.randn(3, 1) y = torch.randn(3, 4) z = y + x.T transposes x to (1,3), making addition incompatible.

3. Final Answer:

   x = torch.randn(3, 1)\ny = torch.randn(3, 4)\nz = x + y -> Option D

4. Quick Check:

   Broadcasting matches trailing dims: 1 and 4 compatible [OK]

Hint: Broadcasting matches shapes from right; 1 can expand to 4 [OK]

Common Mistakes:

• Mixing up tensor shapes and their order
• Assuming transpose always helps broadcasting
• Ignoring dimension order in broadcasting

3. What is the output shape of the following PyTorch code?

import torch
x = torch.randn(2, 3, 1)
y = torch.randn(1, 3, 4)
z = x + y
print(z.shape)

medium

A. torch.Size([2, 3, 4])

B. torch.Size([2, 3, 1])

C. torch.Size([1, 3, 4])

D. RuntimeError due to incompatible shapes

Solution

1. Step 1: Analyze shapes for broadcasting

   x shape is (2,3,1), y shape is (1,3,4). Broadcasting compares dimensions from right: 1 and 4 compatible (1 expands to 4), 3 and 3 match, 2 and 1 compatible (1 expands to 2).

2. Step 2: Determine resulting shape

   Broadcasted shape is (2,3,4) because dimensions expand to the maximum compatible size.

3. Final Answer:

   torch.Size([2, 3, 4]) -> Option A

4. Quick Check:

   Broadcasted shape = max dims per axis [OK]

Hint: Broadcast shape picks max size per dimension from right [OK]

Common Mistakes:

• Ignoring leading dimension broadcasting
• Assuming shapes must be identical
• Thinking 1 cannot expand to larger size

4. Consider this PyTorch code snippet:

import torch
x = torch.randn(4, 1)
y = torch.randn(3, 4)
z = x + y

What is the error and how to fix it?

medium

A. RuntimeError due to shape mismatch; fix by reshaping y to (4,1)

B. RuntimeError due to shape mismatch; fix by reshaping x to (3,1)

C. RuntimeError due to shape mismatch; fix by reshaping x to (1,4)

D. No error; code runs fine

Solution

1. Step 1: Check tensor shapes and broadcasting rules

   x shape is (4,1), y shape is (3,4). Broadcasting compares from right: 1 and 4 compatible (1 expands to 4), 4 and 3 incompatible. So addition fails.

2. Step 2: Fix shapes for broadcasting

   Reshape x to (3,1) so shapes become (3,1) and (3,4). Now 1 and 4 compatible (1 expands to 4), 3 and 3 match. Addition works.

3. Final Answer:

   RuntimeError due to shape mismatch; fix by reshaping x to (3,1) -> Option B

4. Quick Check:

   Broadcast dims must be compatible from right [OK]

Hint: Match trailing dims; reshape smaller tensor to align [OK]

Common Mistakes:

• Trying to reshape y incorrectly
• Ignoring dimension order in broadcasting
• Assuming transpose fixes all shape errors

5. You have a tensor a of shape (5, 1, 3) and a tensor b of shape (1, 4, 1). You want to add them using broadcasting and then compute the mean over the last dimension. Which code snippet correctly does this?

hard

A. c = a + b.T
result = c.mean(dim=2)

B. c = a + b
result = c.mean(dim=1)

C. c = a + b
result = c.mean(dim=2)

D. c = a + b
result = c.mean(dim=0)

Solution

1. Step 1: Verify broadcasting shapes

   a shape: (5,1,3), b shape: (1,4,1). Broadcasting compares dims from right: 3 and 1 compatible (1 expands to 3), 1 and 4 compatible (1 expands to 4), 5 and 1 compatible (1 expands to 5). Result shape is (5,4,3).

2. Step 2: Compute mean over last dimension

   The last dimension is dim=2 (0-based indexing). So mean(dim=2) computes mean over size 3 dimension, resulting in shape (5,4).

3. Final Answer:

   c = a + b\nresult = c.mean(dim=2) -> Option C

4. Quick Check:

   Broadcast shapes then mean over last dim (2) [OK]

Hint: Broadcast shapes, then mean over last dimension (dim=2) [OK]

Common Mistakes:

• Using wrong dimension for mean
• Transposing b unnecessarily
• Confusing dimension indices