Tensor operations (add, mul, matmul) in PyTorch - Cheat Sheet & Quick Revision

Recall & Review

beginner

What does the add operation do on tensors in PyTorch?

The add operation adds corresponding elements of two tensors of the same shape, producing a new tensor with the summed values.

intermediate

Explain the difference between mul and matmul in PyTorch tensor operations.

mul performs element-wise multiplication between two tensors of the same shape, multiplying each element individually. matmul performs matrix multiplication, combining rows and columns according to linear algebra rules.

intermediate

What shape requirements must tensors meet to use matmul in PyTorch?

For matmul, the inner dimensions must match: if tensor A is shape (m, n), tensor B must be shape (n, p). The result will have shape (m, p).

advanced

How does broadcasting work with add or mul in PyTorch?

Broadcasting lets PyTorch automatically expand smaller tensors to match the shape of larger tensors when performing add or mul, as long as their shapes are compatible.

beginner

Write a PyTorch code snippet to multiply two tensors element-wise and then add another tensor.

import torch

x = torch.tensor([1, 2, 3])
y = torch.tensor([4, 5, 6])
z = torch.tensor([7, 8, 9])

result = x.mul(y).add(z)  # element-wise multiply x and y, then add z
print(result)  # tensor([11, 18, 27])

What does torch.matmul(A, B) compute if A is (2, 3) and B is (3, 4)?

A Matrix multiplication resulting in shape (2, 4)

B Element-wise multiplication resulting in shape (2, 3)

C Element-wise addition resulting in shape (2, 3)

D Matrix multiplication resulting in shape (3, 3)

matmul multiplies matrices where inner dimensions match. Here, 3 matches 3, so output shape is (2, 4).

Which operation multiplies tensors element by element?

A matmul

B add

C mul

D dot

mul performs element-wise multiplication between tensors.

If tensor A is shape (5,) and tensor B is shape (1,), what happens when you do A + B in PyTorch?

A Error due to shape mismatch

B Broadcasting adds B to each element of A

C Result is shape (1,)

D Result is shape (5, 1)

PyTorch broadcasts the smaller tensor B to match A's shape and adds element-wise.

What is the output of torch.add(torch.tensor([1,2]), torch.tensor([3,4]))?

A tensor([4, 6])

B tensor([3, 6])

C tensor([1, 2, 3, 4])

D Error

Element-wise addition sums corresponding elements: 1+3=4, 2+4=6.

Which PyTorch function would you use to perform matrix multiplication?

A torch.mul

B torch.add

C torch.sum

D torch.matmul

torch.matmul performs matrix multiplication.

Describe how add, mul, and matmul differ when working with tensors in PyTorch.

Think about how each operation combines tensor elements and their shape rules.

Explain what broadcasting means in the context of tensor operations like add and mul in PyTorch.

Imagine adding a small tensor to a bigger one without manually reshaping.

Practice

1. Which PyTorch function is used for element-wise multiplication of two tensors?

easy

A. torch.matmul()

B. torch.dot()

C. torch.add()

D. torch.mul()

Solution

1. Step 1: Understand element-wise multiplication

   Element-wise multiplication multiplies each element of one tensor by the corresponding element of another tensor.

2. Step 2: Identify the correct PyTorch function

   torch.mul() performs element-wise multiplication, while torch.matmul() is for matrix multiplication.

3. Final Answer:

   torch.mul() -> Option D

4. Quick Check:

   Element-wise multiply = torch.mul() [OK]

Hint: Element-wise multiply uses torch.mul() not matmul [OK]

Common Mistakes:

• Confusing matmul with element-wise multiply
• Using torch.add() instead of multiply
• Trying torch.dot() for element-wise multiply

2. Which of the following is the correct syntax to add two tensors a and b in PyTorch?

easy

A. c = torch.add(a, b)

B. c = a + b + 1

C. c = torch.mul(a, b)

D. c = torch.matmul(a, b)

Solution

1. Step 1: Identify the addition operation

   Adding two tensors element-wise can be done with torch.add(a, b) or a + b.

2. Step 2: Check syntax correctness

   torch.add(a, b) is the correct function call. c = a + b + 1 adds 1 extra, so it's incorrect.

3. Final Answer:

   c = torch.add(a, b) -> Option A

4. Quick Check:

   Tensor add syntax = torch.add(a, b) [OK]

Hint: Use torch.add(a, b) or a + b for tensor addition [OK]

Common Mistakes:

• Adding extra numbers accidentally
• Using mul or matmul instead of add
• Incorrect function call syntax

3. What is the output of the following PyTorch code?

import torch
x = torch.tensor([[1, 2], [3, 4]])
y = torch.tensor([[5, 6], [7, 8]])
z = torch.matmul(x, y)
print(z)

medium

A. tensor([[5, 12], [21, 32]])

B. tensor([[6, 8], [10, 12]])

C. tensor([[19, 22], [43, 50]])

D. tensor([[6, 8], [21, 32]])

Solution

1. Step 1: Perform matrix multiplication manually

   Multiply rows of x by columns of y:
   First row: (1*5 + 2*7) = 19, (1*6 + 2*8) = 22
   Second row: (3*5 + 4*7) = 43, (3*6 + 4*8) = 50

2. Step 2: Confirm output tensor

   The result tensor is [[19, 22], [43, 50]].

3. Final Answer:

   tensor([[19, 22], [43, 50]]) -> Option C

4. Quick Check:

   Matrix multiply result = [[19, 22], [43, 50]] [OK]

Hint: Multiply rows by columns for matmul output [OK]

Common Mistakes:

• Adding elements instead of multiplying
• Confusing element-wise multiply with matmul
• Mixing row and column indices

4. Identify the error in this PyTorch code snippet:

import torch
x = torch.tensor([1, 2, 3])
y = torch.tensor([[4, 5, 6], [7, 8, 9]])
z = torch.matmul(x, y)
print(z)

medium

A. Shape mismatch for matrix multiplication

B. x and y must be the same shape for matmul

C. torch.matmul cannot multiply 1D and 2D tensors

D. No error, code runs fine

Solution

1. Step 1: Check tensor shapes

   x shape is (3,), y shape is (2, 3). For matmul, inner dimensions must match: (N, M) x (M, P).

2. Step 2: Identify mismatch

   Here, x is 1D with length 3, y is 2D with shape (2, 3). The inner dimensions don't align for matmul.

3. Final Answer:

   Shape mismatch for matrix multiplication -> Option A

4. Quick Check:

   Matmul needs matching inner dims = shape mismatch [OK]

Hint: Check tensor shapes before matmul to avoid mismatch [OK]

Common Mistakes:

• Assuming 1D and 2D tensors always multiply
• Ignoring shape rules for matmul
• Thinking matmul works like element-wise multiply

5. Given two tensors a = torch.tensor([[1, 2], [3, 4]]) and b = torch.tensor([[2, 0], [1, 2]]), which code correctly computes the element-wise product and then adds the matrix product of a and b?

hard

A. torch.matmul(torch.mul(a, b), torch.add(a, b))

B. torch.add(torch.mul(a, b), torch.matmul(a, b))

C. torch.mul(torch.add(a, b), torch.matmul(a, b))

D. torch.add(torch.matmul(b, a), torch.mul(a, b))

Solution

1. Step 1: Understand the operations order

   First, compute element-wise product: torch.mul(a, b). Then compute matrix product: torch.matmul(a, b).

2. Step 2: Add the two results

   Use torch.add() to sum the element-wise and matrix products.

3. Final Answer:

   torch.add(torch.mul(a, b), torch.matmul(a, b)) -> Option B

4. Quick Check:

   Element-wise mul + matmul = torch.add(torch.mul(), torch.matmul()) [OK]

Hint: Add element-wise mul and matmul results with torch.add() [OK]

Common Mistakes:

• Mixing order of operations
• Using mul on sums instead of add on products
• Confusing arguments order in add or mul