Tensor shapes and dimensions in PyTorch - Model Metrics & Evaluation

Metrics & Evaluation - Tensor shapes and dimensions

Which metric matters for Tensor shapes and dimensions and WHY

Tensors are like boxes holding numbers in rows, columns, and more directions. The shape tells us how many numbers fit in each direction.

Correct tensor shapes are crucial because models expect inputs and outputs in specific shapes. If shapes mismatch, the model can't learn or predict properly.

So, the key metric here is shape compatibility. It means the input and output tensors must have the right dimensions to fit the model layers.

Confusion matrix or equivalent visualization

Example: A batch of 4 images, each 3 color channels, 28x28 pixels:

Tensor shape: (4, 3, 28, 28)

If a model expects (batch_size, channels, height, width), this fits perfectly.

If you swap height and width or channels, the model will error or give wrong results.

Shape mismatch example:
Input tensor shape: (4, 28, 28, 3)  # channels last instead of channels first
Model expects: (4, 3, 28, 28)

This mismatch causes errors or wrong learning.
    

Tradeoff: Shape flexibility vs. strictness

Sometimes, models can handle flexible shapes (like variable batch size). This helps with memory and speed.

But too much flexibility can cause bugs if shapes don't match layer expectations.

Example:

• Flexible batch size: Good for training with different data amounts.
• Fixed feature size: Must be strict, or model can't learn patterns.

What "good" vs "bad" tensor shapes look like

Good: Shapes match model input/output exactly, e.g., (batch, channels, height, width) for images.

Bad: Shapes mismatch, like missing a dimension or swapping height and width, causing errors or wrong results.

Good shapes ensure smooth training and correct predictions.

Common pitfalls with tensor shapes and dimensions

• Mixing up batch size and feature dimensions.
• Swapping channels last vs. channels first formats.
• Forgetting to flatten tensors before fully connected layers.
• Using wrong shape for labels (e.g., one-hot vs. class indices).
• Ignoring batch dimension, causing shape errors.

Self-check question

Your model expects input shape (batch_size, 3, 32, 32) but your data has shape (batch_size, 32, 32, 3). Is this good? Why or why not?

Answer: No, this is not good. The channel dimension is last in your data but the model expects it second. This mismatch will cause errors or wrong learning.

Key Result

Correct tensor shapes ensure model layers receive data in expected dimensions, preventing errors and enabling proper learning.

Practice

1. What does the shape of a PyTorch tensor represent?

easy

A. The number of elements in each dimension of the tensor

B. The data type of the tensor elements

C. The device where the tensor is stored

D. The memory size of the tensor in bytes

Solution

1. Step 1: Understand tensor shape meaning

   The shape of a tensor tells how many elements are in each dimension, like rows and columns in a matrix.

2. Step 2: Differentiate from other properties

   Data type, device, and memory size are different properties and not related to shape.

3. Final Answer:

   The number of elements in each dimension of the tensor -> Option A

4. Quick Check:

   Tensor shape = elements per dimension [OK]

Hint: Shape = counts of elements per dimension [OK]

Common Mistakes:

• Confusing shape with data type
• Thinking shape shows memory size
• Mixing shape with device location

2. Which of the following is the correct way to get the shape of a tensor t in PyTorch?

easy

A. shape(t)

B. t.shape()

C. t.size()

D. t.get_shape()

Solution

1. Step 1: Recall PyTorch shape methods

   In PyTorch, t.size() returns the shape of the tensor as a tuple-like object.

2. Step 2: Check other options

   t.shape is an attribute, not a method, so t.shape() is incorrect. shape(t) and t.get_shape() are not valid PyTorch syntax.

3. Final Answer:

   t.size() -> Option C

4. Quick Check:

   Use t.size() to get shape [OK]

Hint: Use t.size() method to get shape [OK]

Common Mistakes:

• Calling t.shape() as a method
• Using non-existent functions like shape(t)
• Trying t.get_shape() which is invalid

3. What is the shape of the tensor created by torch.randn(3, 4, 5)?

medium

A. (3, 5, 4)

B. (3, 4, 5)

C. (5, 3, 4)

D. (4, 5, 3)

Solution

1. Step 1: Understand torch.randn arguments

   The arguments (3, 4, 5) specify the size of each dimension in order: first dimension 3, second 4, third 5.

2. Step 2: Confirm shape matches arguments

   The tensor shape matches exactly the arguments given, so the shape is (3, 4, 5).

3. Final Answer:

   (3, 4, 5) -> Option B

4. Quick Check:

   Shape matches input sizes (3,4,5) [OK]

Hint: Shape matches torch.randn arguments order [OK]

Common Mistakes:

• Mixing up dimension order
• Assuming shape is reversed
• Confusing shape with total elements

4. Consider this code snippet:

import torch
t = torch.randn(2, 3)
print(t.shape())

What error will this code produce?

medium

A. TypeError: 'torch.Size' object is not callable

B. AttributeError: 'Tensor' object has no attribute 'shape'

C. SyntaxError: invalid syntax

D. No error, prints (2, 3)

Solution

1. Step 1: Identify usage of t.shape()

   In PyTorch, t.shape is an attribute, not a method, so calling t.shape() tries to call a non-callable object.

2. Step 2: Understand error type

   This causes a TypeError saying the 'torch.Size' object is not callable.

3. Final Answer:

   TypeError: 'torch.Size' object is not callable -> Option A

4. Quick Check:

   t.shape is attribute, not method [OK]

Hint: t.shape is attribute, don't add () [OK]

Common Mistakes:

• Calling t.shape() instead of t.shape
• Expecting no error from t.shape()
• Confusing AttributeError with TypeError

5. You have a tensor x with shape (10, 5, 3). You want to reshape it to a 2D tensor with 5 rows and keep the total number of elements the same. Which of the following shapes is valid for the reshaped tensor?

hard

A. (5, 15)

B. (5, 10, 3)

C. (5, 5, 6)

D. (5, 30)

Solution

1. Step 1: Calculate total elements in original tensor

   Original shape is (10, 5, 3), so total elements = 10 * 5 * 3 = 150.

2. Step 2: Check which new shape keeps total elements and has 5 rows

   We want 5 rows, so first dimension = 5. The second dimension must satisfy 5 * ? = 150, so ? = 30. Check options:
   - A: (5, 15) -> 5*15=75 invalid total elements
   - B: (5, 10, 3) -> 5*10*3=150 valid but has 3 dims, not 2
   - C: (5, 30) -> 5*30=150 valid
   - D: (5, 5, 6) -> 5*5*6=150 valid but 3 dims

3. Step 3: Confirm valid shape with 5 rows and total elements

   (5, 30) is (5, 30) which matches 5 rows and total elements 150. (5, 15) is incorrect because 5*15=75, less than 150.

4. Final Answer:

   (5, 30) -> Option D

5. Quick Check:

   5 * 30 = 150 elements [OK]

Hint: Total elements must stay same; multiply dims to check [OK]

Common Mistakes:

• Ignoring total element count when reshaping
• Choosing shape with wrong total elements
• Confusing number of dimensions required