Augmentation policy search (AutoAugment) in Computer Vision - Model Metrics & Evaluation

Metrics & Evaluation - Augmentation policy search (AutoAugment)

Which metric matters for Augmentation policy search (AutoAugment) and WHY

AutoAugment helps improve image model training by finding the best ways to change images. The key metric to watch is validation accuracy. This shows if the model learns better with new image styles. Higher accuracy means the augmentation helps the model see more useful patterns and generalize well to new images.

Confusion matrix or equivalent visualization

Confusion Matrix Example (for image classification):

          Predicted
          Cat  Dog  Bird
Actual Cat  45   3    2
       Dog   4   40    6
       Bird  1    5   44

Total samples = 150

From this, accuracy = (45+40+44)/150 = 0.86 (86%)

AutoAugment aims to improve this accuracy by better training data.
    

Precision vs Recall tradeoff with concrete examples

In image classification, precision and recall matter per class. For example, if the model detects cats:

• Precision means: Of all images predicted as cats, how many really are cats?
• Recall means: Of all actual cat images, how many did the model find?

AutoAugment helps balance these by making the model see varied images. For example, if recall is low, the model misses many cats. Augmentation can add more cat-like images to fix this. If precision is low, the model confuses cats with dogs. Augmentation can help by showing clearer differences.

What "good" vs "bad" metric values look like for this use case

Good: Validation accuracy above 85% with balanced precision and recall near 0.8 or higher per class. This means the model correctly identifies most images and makes few mistakes.

Bad: Validation accuracy below 70%, or precision/recall below 0.5 for important classes. This means the model struggles to learn useful features, possibly due to poor augmentation or overfitting.

Metrics pitfalls

• Accuracy paradox: High accuracy can hide poor class performance if classes are imbalanced.
• Data leakage: Augmentation must be applied only on training data, not validation, or metrics will be misleading.
• Overfitting indicators: Training accuracy much higher than validation accuracy means augmentation may not be effective or model is too complex.

Self-check question

Your image classifier trained with AutoAugment has 98% training accuracy but only 60% validation accuracy. Is this good? Why or why not?

Answer: This is not good. The large gap shows overfitting. The model memorizes training images but fails to generalize. AutoAugment should help reduce this gap by making training images more varied. You may need to adjust augmentation policies or model complexity.

Key Result

Validation accuracy is key to measure if AutoAugment improves model generalization by providing better training images.

Practice

1. What is the main purpose of AutoAugment in computer vision?

easy

A. To convert images into grayscale before training

B. To reduce the size of the training dataset

C. To increase the number of layers in a neural network

D. To automatically find the best image transformations to improve model training

Solution

1. Step 1: Understand AutoAugment's goal

   AutoAugment searches for the best image changes (augmentations) to help models learn better.

2. Step 2: Compare options to this goal

   Only To automatically find the best image transformations to improve model training describes this automatic search for image transformations to improve training.

3. Final Answer:

   To automatically find the best image transformations to improve model training -> Option D

4. Quick Check:

   AutoAugment = automatic image transformation search [OK]

Hint: AutoAugment = automatic image changes for better training [OK]

Common Mistakes:

• Thinking AutoAugment reduces dataset size
• Confusing AutoAugment with model architecture changes
• Assuming it only converts images to grayscale

2. Which of the following is the correct way to apply AutoAugment using TensorFlow Addons in Python?

easy

A. tf.image.auto_augment(image)

B. tfa.image.auto_augment(image)

C. tfa.auto_augment(image)

D. tf.autoaugment(image)

Solution

1. Step 1: Identify the correct library and function

   TensorFlow Addons provides AutoAugment under the image module as tfa.image.auto_augment.

2. Step 2: Check each option's syntax

   Only tfa.image.auto_augment(image) uses the correct module and function name: tfa.image.auto_augment(image).

3. Final Answer:

   tfa.image.auto_augment(image) -> Option B

4. Quick Check:

   TensorFlow Addons image module = tfa.image.auto_augment [OK]

Hint: AutoAugment is in tfa.image, not tf or tfa root [OK]

Common Mistakes:

• Using tf.autoaugment instead of tfa.image.auto_augment
• Missing the image submodule in tfa
• Incorrect function name like auto_augment vs autoaugment

3. Given the following code snippet using AutoAugment, what will be the output shape of the augmented image?

import tensorflow as tf
import tensorflow_addons as tfa

image = tf.random.uniform([32, 32, 3])
augmented_image = tfa.image.auto_augment(image)
print(augmented_image.shape)

medium

A. (32, 32, 3)

B. (None, None, 3)

C. (32, 32)

D. (3, 32, 32)

Solution

1. Step 1: Check input image shape

   The input image has shape (32, 32, 3), meaning height=32, width=32, and 3 color channels.

2. Step 2: Understand AutoAugment output shape

   AutoAugment applies transformations but keeps the original image size and channels unchanged.

3. Final Answer:

   (32, 32, 3) -> Option A

4. Quick Check:

   AutoAugment keeps image shape same [OK]

Hint: AutoAugment changes pixels, not image size [OK]

Common Mistakes:

• Assuming output shape changes after augmentation
• Confusing channel order or missing channels
• Expecting batch dimension in output shape

4. You wrote this code to apply AutoAugment but get an error: AttributeError: module 'tensorflow_addons' has no attribute 'auto_augment'. What is the most likely fix?

medium

A. Import the image submodule and use tfa.image.auto_augment instead

B. Update TensorFlow to the latest version

C. Use tf.image.auto_augment instead of tfa

D. Restart the Python kernel

Solution

1. Step 1: Understand the error cause

   The error says tfa.auto_augment does not exist, meaning the function is not at the root of tensorflow_addons.

2. Step 2: Correct usage of AutoAugment in TensorFlow Addons

   The correct function is inside the image submodule: tfa.image.auto_augment.

3. Final Answer:

   Import the image submodule and use tfa.image.auto_augment instead -> Option A

4. Quick Check:

   Use tfa.image.auto_augment, not tfa.auto_augment [OK]

Hint: AutoAugment is in tfa.image, not tfa root [OK]

Common Mistakes:

• Trying to use tfa.auto_augment directly
• Assuming tf.image has AutoAugment
• Ignoring the need to import submodules

5. You want to improve your image classification model by using AutoAugment policies. Which approach best combines AutoAugment with training to maximize accuracy?

hard

A. Use AutoAugment only on validation images to test robustness

B. Apply AutoAugment once to the entire training dataset before training

C. Apply AutoAugment transformations on training images dynamically during each training epoch

D. Skip AutoAugment and increase model layers instead

Solution

1. Step 1: Understand dynamic augmentation benefits

   Applying AutoAugment dynamically during training shows varied images each epoch, helping the model generalize better.

2. Step 2: Compare static vs dynamic application

   Applying once (Apply AutoAugment once to the entire training dataset before training) limits variety; applying on validation (Use AutoAugment only on validation images to test robustness) is incorrect; increasing layers (Skip AutoAugment and increase model layers instead) is unrelated.

3. Final Answer:

   Apply AutoAugment transformations on training images dynamically during each training epoch -> Option C

4. Quick Check:

   Dynamic augmentation during training = best accuracy [OK]

Hint: Augment images freshly each epoch for best learning [OK]

Common Mistakes:

• Applying augmentation only once before training
• Augmenting validation data instead of training data
• Ignoring augmentation and changing model size