Fine-tuning approach in TensorFlow - Model Metrics & Evaluation

Metrics & Evaluation - Fine-tuning approach

Which metric matters for Fine-tuning and WHY

Fine-tuning means adjusting a pre-trained model to a new task. The best metric depends on the task type:

• Classification: Accuracy, Precision, Recall, and F1-score matter to check if the model correctly labels new data.
• Regression: Mean Squared Error (MSE) or Mean Absolute Error (MAE) show how close predictions are to real values.

Because fine-tuning often uses small new data, overfitting is a risk. So, monitoring validation loss and validation accuracy helps know if the model learns well without memorizing.

Confusion Matrix Example for Fine-tuned Classification

Imagine a fine-tuned model classifying cats vs dogs. Here is a confusion matrix after testing 100 images:

      | Predicted Cat | Predicted Dog |
      |---------------|---------------|
      | True Cat: 40  | False Dog: 5  |
      | False Cat: 5  | True Dog: 50  |
    

From this:

• True Positives (TP) = 40 (cats correctly identified)
• False Positives (FP) = 5 (dogs wrongly called cats)
• True Negatives (TN) = 50 (dogs correctly identified)
• False Negatives (FN) = 5 (cats missed)

Precision vs Recall Tradeoff in Fine-tuning

Fine-tuning can change how the model balances precision and recall:

• Precision = TP / (TP + FP): How many predicted positives are correct?
• Recall = TP / (TP + FN): How many actual positives did we find?

Example: For a medical image model fine-tuned to detect disease:

• High recall is critical to catch all sick patients (avoid missing any).
• High precision avoids false alarms (not telling healthy people they are sick).

Fine-tuning can adjust this balance by changing training data or layers trained.

Good vs Bad Metric Values for Fine-tuning

Good fine-tuning results show:

• Validation accuracy close to or better than training accuracy (no big gap).
• Precision and recall both high (above 0.8) for balanced tasks.
• Loss decreasing steadily on training and validation sets.

Bad signs include:

• Validation accuracy much lower than training (overfitting).
• Precision very high but recall very low, or vice versa (unbalanced).
• Validation loss increasing while training loss decreases.

Common Metric Pitfalls in Fine-tuning

• Accuracy paradox: High accuracy can be misleading if classes are imbalanced.
• Data leakage: Using test data during fine-tuning inflates metrics falsely.
• Overfitting: Fine-tuning on small data can cause the model to memorize, not generalize.
• Ignoring validation metrics: Only looking at training metrics hides poor real-world performance.

Self-Check Question

Your fine-tuned model has 98% accuracy but only 12% recall on the positive class (e.g., fraud detection). Is it good for production? Why or why not?

Answer: No, it is not good. The low recall means the model misses most positive cases (fraud). Even with high accuracy, it fails the main goal of catching fraud. You should improve recall before using it.

Key Result

Fine-tuning success depends on balanced metrics like precision and recall, and avoiding overfitting by monitoring validation performance.

Practice

1. What is the main purpose of fine-tuning a pretrained TensorFlow model?

easy

A. To adapt the model to a new task by training some layers

B. To train a model from scratch with random weights

C. To increase the size of the dataset automatically

D. To reduce the number of layers in the model

Solution

1. Step 1: Understand fine-tuning concept

   Fine-tuning means adjusting a pretrained model to work well on a new task by training some of its layers.

2. Step 2: Compare options

   Only To adapt the model to a new task by training some layers correctly describes this process. Other options describe unrelated tasks.

3. Final Answer:

   To adapt the model to a new task by training some layers -> Option A

4. Quick Check:

   Fine-tuning = adapt pretrained model [OK]

Hint: Fine-tuning means training some layers on new data [OK]

Common Mistakes:

• Thinking fine-tuning means training from scratch
• Confusing fine-tuning with data augmentation
• Assuming fine-tuning reduces model size

2. Which TensorFlow code snippet correctly freezes the base model layers before fine-tuning?

easy

A. base_model.layers.freeze = True

B. base_model.trainable = False

C. base_model.freeze()

D. base_model.trainable = True

Solution

1. Step 1: Recall TensorFlow syntax for freezing layers

   In TensorFlow, setting trainable = False freezes the layers.

2. Step 2: Check options

   base_model.trainable = False uses correct syntax. Options A, B, and C are invalid methods/attributes or incorrect (e.g., trainable = True unfreezes layers).

3. Final Answer:

   base_model.trainable = False -> Option B

4. Quick Check:

   Freeze layers = trainable False [OK]

Hint: Freeze layers by setting trainable to False [OK]

Common Mistakes:

• Using incorrect method names like freeze()
• Setting trainable to True to freeze layers
• Trying to freeze layers by modifying layer attributes directly

3. What will be the output shape of the model after adding a new Dense layer with 10 units on top of a base model output shape (None, 1280)?

base_model = tf.keras.applications.MobileNetV2(input_shape=(224,224,3), include_top=False, pooling='avg')
new_output = tf.keras.layers.Dense(10, activation='softmax')(base_model.output)
model = tf.keras.Model(inputs=base_model.input, outputs=new_output)
print(model.output_shape)

medium

A. (None, 128)

B. (None, 224, 224, 3)

C. (None, 10)

D. (10, None)

Solution

1. Step 1: Understand base model output shape

   The base model outputs a tensor with shape (None, 1280) due to pooling='avg'.

2. Step 2: Adding Dense layer changes output shape

   The Dense layer with 10 units changes output shape to (None, 10), where None is batch size.

3. Final Answer:

   (None, 10) -> Option C

4. Quick Check:

   Dense units = output shape last dim [OK]

Hint: Dense layer units define output shape's last dimension [OK]

Common Mistakes:

• Confusing input shape with output shape
• Ignoring batch size dimension None
• Assuming output shape keeps base model shape

4. Identify the error in this fine-tuning code snippet:

base_model = tf.keras.applications.ResNet50(weights='imagenet', include_top=False)
base_model.trainable = False

model = tf.keras.Sequential([
    base_model,
    tf.keras.layers.Dense(5, activation='softmax')
])

model.compile(optimizer='adam', loss='sparse_categorical_crossentropy')
model.fit(train_data, epochs=5)

base_model.trainable = True
model.fit(train_data, epochs=5, learning_rate=0.001)

medium

A. Dense layer should be added before base_model

B. base_model.trainable should be True before first fit

C. Loss function is incorrect for classification

D. learning_rate is not a valid argument in model.fit

Solution

1. Step 1: Check model.fit arguments

   The model.fit method does not accept learning_rate as an argument; it belongs to the optimizer.

2. Step 2: Identify correct way to change learning rate

   To change learning rate, create a new optimizer with desired learning rate and recompile the model.

3. Final Answer:

   learning_rate is not a valid argument in model.fit -> Option D

4. Quick Check:

   learning_rate belongs to optimizer, not fit [OK]

Hint: learning_rate must be set in optimizer, not fit() [OK]

Common Mistakes:

• Passing learning_rate to model.fit
• Not recompiling model after changing trainable layers
• Freezing layers after first training instead of before

5. You want to fine-tune a pretrained model on a small dataset. Which approach is best to avoid overfitting?

# Step 1: Freeze base model
base_model.trainable = False
# Step 2: Add new classifier layers
# Step 3: Train new layers
# Step 4: Unfreeze some base layers
# Step 5: Continue training with low learning rate

Which of these is the best practice for Step 5?

hard

A. Use a very low learning rate like 1e-5 to fine-tune base layers

B. Use a high learning rate like 0.1 to speed up training

C. Keep learning rate same as initial training

D. Stop training immediately after unfreezing layers

Solution

1. Step 1: Understand fine-tuning with small data

   Fine-tuning with small data risks overfitting, so training must be gentle.

2. Step 2: Choose learning rate carefully

   A very low learning rate (e.g., 1e-5) helps adjust pretrained weights slowly to avoid overfitting.

3. Final Answer:

   Use a very low learning rate like 1e-5 to fine-tune base layers -> Option A

4. Quick Check:

   Low learning rate prevents overfitting in fine-tuning [OK]

Hint: Use low learning rate when unfreezing pretrained layers [OK]

Common Mistakes:

• Using high learning rate causing weight damage
• Not unfreezing any layers for fine-tuning
• Stopping training too early after unfreezing