Custom transforms in PyTorch - Model Metrics & Evaluation

Metrics & Evaluation - Custom transforms

Which metric matters for Custom Transforms and WHY

Custom transforms change data before training. The main goal is to improve model learning by making data clearer or more varied.

Metrics to watch are training loss and validation accuracy. Lower loss means the model fits data well. Higher accuracy means the model predicts better on new data.

If transforms help, validation accuracy should improve without overfitting (training accuracy too high but validation low).

Confusion Matrix Example

Suppose we classify images after applying custom transforms. Here is a confusion matrix:

      | Predicted Positive | Predicted Negative |
      |--------------------|--------------------|
      | True Positive (TP): 80 | False Negative (FN): 20 |
      | False Positive (FP): 10 | True Negative (TN): 90 |
    

Total samples = 80 + 20 + 10 + 90 = 200

From this, we calculate:

• Precision = TP / (TP + FP) = 80 / (80 + 10) = 0.89
• Recall = TP / (TP + FN) = 80 / (80 + 20) = 0.80
• F1 Score = 2 * (Precision * Recall) / (Precision + Recall) ≈ 0.84

Precision vs Recall Tradeoff with Custom Transforms

Custom transforms can affect precision and recall differently.

For example, if transforms add noise, recall might drop because the model misses some true positives.

If transforms sharpen features, precision might improve because the model makes fewer false positive mistakes.

Choosing transforms depends on the task:

• For medical images, high recall is key (catch all cases).
• For spam detection, high precision is important (avoid marking good emails as spam).

Good vs Bad Metric Values for Custom Transforms

Good:

• Validation accuracy improves or stays stable after adding transforms.
• Training loss decreases steadily.
• Precision and recall both improve or balance well.

Bad:

• Validation accuracy drops significantly.
• Training loss stays high or fluctuates.
• Precision or recall drops sharply, indicating transforms confuse the model.

Common Pitfalls with Metrics and Custom Transforms

• Accuracy paradox: Accuracy looks good but model fails on minority classes.
• Data leakage: Transforms accidentally use test data info, inflating metrics.
• Overfitting: Transforms make training data too easy, causing poor generalization.
• Ignoring validation: Only training metrics improve, but validation worsens.

Self Check

Your model has 98% accuracy but 12% recall on fraud cases after applying custom transforms. Is it good?

Answer: No. The low recall means the model misses most fraud cases, which is critical. High accuracy is misleading because fraud is rare. You should improve recall, possibly by adjusting transforms or model.

Key Result

Custom transforms should improve validation accuracy and balance precision and recall to help the model learn better.

Practice

1. What is the main purpose of creating a custom transform in PyTorch?

easy

A. To speed up model training by changing the optimizer

B. To save the model after training

C. To define a new neural network layer

D. To apply a specific data change not available in standard transforms

Solution

1. Step 1: Understand what transforms do

   Transforms change input data before training to help the model learn better.

2. Step 2: Identify the role of custom transforms

   Custom transforms let you add new ways to change data that standard tools don't have.

3. Final Answer:

   To apply a specific data change not available in standard transforms -> Option D

4. Quick Check:

   Custom transform = new data change [OK]

Hint: Custom transforms change data uniquely before training [OK]

Common Mistakes:

• Confusing transforms with model layers
• Thinking transforms speed up training directly
• Mixing transforms with saving models

2. Which of the following is the correct way to define a custom transform class in PyTorch?

easy

A. class MyTransform: def transform(self, x): return x * 2

B. class MyTransform: def __call__(self, x): return x * 2

C. def MyTransform(x): return x * 2

D. def MyTransform: return x * 2

Solution

1. Step 1: Recall custom transform structure

   Custom transforms are classes with a __call__ method to make them callable.

2. Step 2: Check each option

   class MyTransform: def __call__(self, x): return x * 2 defines a class with __call__, correct syntax. Others miss class or __call__ or have wrong syntax.

3. Final Answer:

   class MyTransform: def __call__(self, x): return x * 2 -> Option B

4. Quick Check:

   Custom transform = class + __call__ method [OK]

Hint: Custom transform must be a class with __call__ method [OK]

Common Mistakes:

• Defining transform as a function only
• Using wrong method name like transform instead of __call__
• Missing class keyword

3. Given this custom transform code, what will be the output of transform(3)?

class AddFive:
    def __call__(self, x):
        return x + 5

transform = AddFive()

medium

A. 8

B. 15

C. 3

D. Error

Solution

1. Step 1: Understand the transform class behavior

   The __call__ method adds 5 to the input x.

2. Step 2: Calculate transform(3)

   Input 3 plus 5 equals 8.

3. Final Answer:

   8 -> Option A

4. Quick Check:

   3 + 5 = 8 [OK]

Hint: Call __call__ method adds 5 to input [OK]

Common Mistakes:

• Confusing addition with multiplication
• Expecting the original input unchanged
• Thinking transform is not callable

4. Identify the error in this custom transform code:

class Multiply:
    def call(self, x):
        return x * 3

transform = Multiply()
print(transform(4))

medium

A. Method should be named __call__, not call

B. Class should inherit from nn.Module

C. Missing return statement in method

D. transform(4) is invalid syntax

Solution

1. Step 1: Check method name for callable class

   To make an instance callable, method must be __call__, not call.

2. Step 2: Verify other parts

   Class inheritance is optional for transforms; return is present; calling instance is valid.

3. Final Answer:

   Method should be named __call__, not call -> Option A

4. Quick Check:

   Callable class needs __call__ method [OK]

Hint: Callable classes need __call__, not call method [OK]

Common Mistakes:

• Using call instead of __call__
• Thinking inheritance from nn.Module is required
• Missing return statement

5. You want to create a custom transform that normalizes a tensor by subtracting its mean and dividing by its standard deviation. Which code correctly implements this?

hard

A. class Normalize:
def __call__(self, x):
return x + x.mean() + x.std()

B. class Normalize:
def __call__(self, x):
return x / x.mean() - x.std()

C. class Normalize:
def __call__(self, x):
return (x - x.mean()) / x.std()

D. class Normalize:
def __call__(self, x):
return x * x.mean() / x.std()

Solution

1. Step 1: Recall normalization formula

   Normalization means subtract mean and divide by standard deviation.

2. Step 2: Check each option's formula

   class Normalize: def __call__(self, x): return (x - x.mean()) / x.std() matches formula: (x - mean) / std. Others do incorrect math.

3. Final Answer:

   class Normalize: def __call__(self, x): return (x - x.mean()) / x.std() -> Option C

4. Quick Check:

   Normalize = (x - mean) / std [OK]

Hint: Normalize = subtract mean, then divide by std [OK]

Common Mistakes:

• Dividing before subtracting mean
• Adding mean and std instead of subtract/divide
• Multiplying by mean/std incorrectly