Tensor operations (add, mul, matmul) in PyTorch - Model Metrics & Evaluation

Metrics & Evaluation - Tensor operations (add, mul, matmul)

Which metric matters for Tensor operations and WHY

When working with tensor operations like add, multiply, and matrix multiply, the key metric is correctness. This means the output tensor values must be exactly what math says they should be. For example, adding two tensors should give a tensor where each element is the sum of the corresponding elements. Multiplying element-wise should multiply each element pair. Matrix multiplication should follow the rules of linear algebra.

Correctness matters because these operations are the building blocks of machine learning models. If the math is wrong, the model will learn wrong patterns or crash.

Confusion matrix or equivalent visualization

For tensor operations, we don't use confusion matrices. Instead, we check outputs by comparing expected and actual tensors.

Input A:       Input B:
[[1, 2],     [[3, 4],
 [5, 6]]      [7, 8]]

Add (A + B):
[[4, 6],
 [12, 14]]

Mul (A * B) element-wise:
[[3, 8],
 [35, 48]]

Matmul (A @ B):
[[17, 20],
 [53, 62]]
    

We verify that the output matches these expected results exactly.

Tradeoff: Speed vs Precision in Tensor Operations

Sometimes, faster tensor operations use less precise numbers (like float16 instead of float32). This speeds up training but can cause small errors.

If you want very accurate results (like in scientific computing), use higher precision. If you want faster training and can accept tiny errors, use lower precision.

Choosing the right balance depends on your task and hardware.

What good vs bad tensor operation results look like

Good: Output tensors exactly match expected values from math. No unexpected NaNs or infinities. Shapes are correct.

Bad: Outputs have wrong values, shapes don't match, or contain NaNs/Infs. This means a bug or hardware issue.

Common pitfalls with tensor operations

• Shape mismatch errors when adding or multiplying tensors of incompatible sizes.
• Confusing element-wise multiplication (mul) with matrix multiplication (matmul).
• Using wrong data types causing precision loss or overflow.
• Forgetting broadcasting rules, leading to unexpected results.
• Not checking for NaNs or Infs after operations, which can break training.

Self-check question

Your tensor addition outputs values that are off by a small amount (like 1e-5). Is this a problem?

Answer: Usually, small floating point differences are normal due to how computers handle decimals. But if differences are large or cause errors, then it is a problem.

Key Result

Correctness of tensor outputs (matching expected math results) is the key metric for tensor operations.

Practice

1. Which PyTorch function is used for element-wise multiplication of two tensors?

easy

A. torch.matmul()

B. torch.dot()

C. torch.add()

D. torch.mul()

Solution

1. Step 1: Understand element-wise multiplication

   Element-wise multiplication multiplies each element of one tensor by the corresponding element of another tensor.

2. Step 2: Identify the correct PyTorch function

   torch.mul() performs element-wise multiplication, while torch.matmul() is for matrix multiplication.

3. Final Answer:

   torch.mul() -> Option D

4. Quick Check:

   Element-wise multiply = torch.mul() [OK]

Hint: Element-wise multiply uses torch.mul() not matmul [OK]

Common Mistakes:

• Confusing matmul with element-wise multiply
• Using torch.add() instead of multiply
• Trying torch.dot() for element-wise multiply

2. Which of the following is the correct syntax to add two tensors a and b in PyTorch?

easy

A. c = torch.add(a, b)

B. c = a + b + 1

C. c = torch.mul(a, b)

D. c = torch.matmul(a, b)

Solution

1. Step 1: Identify the addition operation

   Adding two tensors element-wise can be done with torch.add(a, b) or a + b.

2. Step 2: Check syntax correctness

   torch.add(a, b) is the correct function call. c = a + b + 1 adds 1 extra, so it's incorrect.

3. Final Answer:

   c = torch.add(a, b) -> Option A

4. Quick Check:

   Tensor add syntax = torch.add(a, b) [OK]

Hint: Use torch.add(a, b) or a + b for tensor addition [OK]

Common Mistakes:

• Adding extra numbers accidentally
• Using mul or matmul instead of add
• Incorrect function call syntax

3. What is the output of the following PyTorch code?

import torch
x = torch.tensor([[1, 2], [3, 4]])
y = torch.tensor([[5, 6], [7, 8]])
z = torch.matmul(x, y)
print(z)

medium

A. tensor([[5, 12], [21, 32]])

B. tensor([[6, 8], [10, 12]])

C. tensor([[19, 22], [43, 50]])

D. tensor([[6, 8], [21, 32]])

Solution

1. Step 1: Perform matrix multiplication manually

   Multiply rows of x by columns of y:
   First row: (1*5 + 2*7) = 19, (1*6 + 2*8) = 22
   Second row: (3*5 + 4*7) = 43, (3*6 + 4*8) = 50

2. Step 2: Confirm output tensor

   The result tensor is [[19, 22], [43, 50]].

3. Final Answer:

   tensor([[19, 22], [43, 50]]) -> Option C

4. Quick Check:

   Matrix multiply result = [[19, 22], [43, 50]] [OK]

Hint: Multiply rows by columns for matmul output [OK]

Common Mistakes:

• Adding elements instead of multiplying
• Confusing element-wise multiply with matmul
• Mixing row and column indices

4. Identify the error in this PyTorch code snippet:

import torch
x = torch.tensor([1, 2, 3])
y = torch.tensor([[4, 5, 6], [7, 8, 9]])
z = torch.matmul(x, y)
print(z)

medium

A. Shape mismatch for matrix multiplication

B. x and y must be the same shape for matmul

C. torch.matmul cannot multiply 1D and 2D tensors

D. No error, code runs fine

Solution

1. Step 1: Check tensor shapes

   x shape is (3,), y shape is (2, 3). For matmul, inner dimensions must match: (N, M) x (M, P).

2. Step 2: Identify mismatch

   Here, x is 1D with length 3, y is 2D with shape (2, 3). The inner dimensions don't align for matmul.

3. Final Answer:

   Shape mismatch for matrix multiplication -> Option A

4. Quick Check:

   Matmul needs matching inner dims = shape mismatch [OK]

Hint: Check tensor shapes before matmul to avoid mismatch [OK]

Common Mistakes:

• Assuming 1D and 2D tensors always multiply
• Ignoring shape rules for matmul
• Thinking matmul works like element-wise multiply

5. Given two tensors a = torch.tensor([[1, 2], [3, 4]]) and b = torch.tensor([[2, 0], [1, 2]]), which code correctly computes the element-wise product and then adds the matrix product of a and b?

hard

A. torch.matmul(torch.mul(a, b), torch.add(a, b))

B. torch.add(torch.mul(a, b), torch.matmul(a, b))

C. torch.mul(torch.add(a, b), torch.matmul(a, b))

D. torch.add(torch.matmul(b, a), torch.mul(a, b))

Solution

1. Step 1: Understand the operations order

   First, compute element-wise product: torch.mul(a, b). Then compute matrix product: torch.matmul(a, b).

2. Step 2: Add the two results

   Use torch.add() to sum the element-wise and matrix products.

3. Final Answer:

   torch.add(torch.mul(a, b), torch.matmul(a, b)) -> Option B

4. Quick Check:

   Element-wise mul + matmul = torch.add(torch.mul(), torch.matmul()) [OK]

Hint: Add element-wise mul and matmul results with torch.add() [OK]

Common Mistakes:

• Mixing order of operations
• Using mul on sums instead of add on products
• Confusing arguments order in add or mul