Freezing and unfreezing layers in TensorFlow - Model Metrics & Evaluation

Metrics & Evaluation - Freezing and unfreezing layers

Which metric matters for Freezing and Unfreezing Layers and WHY

When you freeze layers in a model, you keep some parts fixed and only train others. This helps when you have a small dataset or want to keep learned features. The key metrics to watch are validation loss and validation accuracy. They show if the model is learning new useful patterns without forgetting old ones.

If validation accuracy improves and validation loss decreases after unfreezing layers, it means the model is adapting well. If validation loss goes up or accuracy drops, the model might be overfitting or forgetting.

Confusion Matrix Example

Suppose you use a frozen base model and train a new classifier on top. After unfreezing some layers, you test on 100 samples and get:

      | Predicted Positive | Predicted Negative |
      |--------------------|--------------------|
      | True Positive (TP) = 40 | False Negative (FN) = 10 |
      | False Positive (FP) = 5  | True Negative (TN) = 45 |
    

Total samples = 40 + 10 + 5 + 45 = 100

From this, you calculate:

• Precision = TP / (TP + FP) = 40 / (40 + 5) = 0.89
• Recall = TP / (TP + FN) = 40 / (40 + 10) = 0.80
• Accuracy = (TP + TN) / Total = (40 + 45) / 100 = 0.85

Precision vs Recall Tradeoff in Freezing and Unfreezing Layers

When you unfreeze layers, the model can learn more details but risks overfitting. This can affect precision and recall differently:

• High Precision, Low Recall: The model is very sure about positive predictions but misses many actual positives. This can happen if unfreezing causes the model to be too strict.
• High Recall, Low Precision: The model finds most positives but also makes many false alarms. This can happen if unfreezing makes the model too sensitive.

Choosing how many layers to unfreeze balances this tradeoff. Start by freezing most layers, then gradually unfreeze more if validation metrics improve.

Good vs Bad Metric Values for Freezing and Unfreezing Layers

Good:

• Validation accuracy steadily improves or stays stable after unfreezing.
• Validation loss decreases or remains low, showing better generalization.
• Precision and recall both improve or stay balanced.

Bad:

• Validation accuracy drops after unfreezing, indicating overfitting or forgetting.
• Validation loss increases, showing the model struggles to generalize.
• Precision or recall becomes very low, meaning the model is biased or unstable.

Common Pitfalls in Metrics When Freezing and Unfreezing Layers

• Overfitting: Unfreezing too many layers on a small dataset can cause the model to memorize training data, leading to low validation performance.
• Data Leakage: If validation data leaks into training, metrics look better but don't reflect real performance.
• Ignoring Validation Metrics: Only watching training loss can mislead; always check validation loss and accuracy.
• Sudden Metric Drops: Unfreezing all layers at once can cause unstable training and metric drops.

Self Check

Your model has 98% accuracy but 12% recall on fraud detection after unfreezing layers. Is it good for production?

Answer: No. High accuracy is misleading because fraud cases are rare. Low recall means the model misses most frauds, which is dangerous. You should improve recall even if accuracy drops.

Key Result

Validation accuracy and loss are key to check if unfreezing layers improves learning without overfitting.

Practice

1. What does freezing layers in a TensorFlow model do?

easy

A. Stops the layers from updating their weights during training

B. Allows the layers to learn new features

C. Deletes the layers from the model

D. Increases the learning rate for those layers

Solution

1. Step 1: Understand the meaning of freezing layers

   Freezing layers means preventing them from changing their weights during training.

2. Step 2: Identify the effect of freezing

   Since weights do not update, the layers keep their learned knowledge intact.

3. Final Answer:

   Stops the layers from updating their weights during training -> Option A

4. Quick Check:

   Freezing = stop weight updates [OK]

Hint: Freezing means no weight updates during training [OK]

Common Mistakes:

• Thinking freezing deletes layers
• Believing freezing increases learning rate
• Confusing freezing with unfreezing

2. Which is the correct way to freeze a layer named layer in TensorFlow before compiling the model?

easy

A. layer.frozen = True

B. layer.freeze = True

C. layer.trainable = True

D. layer.trainable = False

Solution

1. Step 1: Recall the attribute to control training

   TensorFlow uses the trainable attribute to allow or prevent training on layers.

2. Step 2: Set the attribute to freeze

   Setting trainable = False freezes the layer so it won't update during training.

3. Final Answer:

   layer.trainable = False -> Option D

4. Quick Check:

   Freeze layer = trainable False [OK]

Hint: Use trainable = False to freeze layers before compile [OK]

Common Mistakes:

• Using non-existent attributes like freeze or frozen
• Setting trainable = True to freeze
• Changing trainable after compiling

3. Given the code below, what will be the output of model.layers[0].trainable after compiling?

model.layers[0].trainable = False
model.compile(optimizer='adam', loss='mse')
print(model.layers[0].trainable)

medium

A. None

B. True

C. False

D. Raises an error

Solution

1. Step 1: Set layer.trainable before compiling

   The code sets trainable = False before calling compile.

2. Step 2: Check if trainable attribute changes after compile

   The trainable attribute remains as set before compile; it does not change automatically.

3. Final Answer:

   False -> Option C

4. Quick Check:

   trainable stays False after compile [OK]

Hint: Set trainable before compile; it stays unchanged after [OK]

Common Mistakes:

• Assuming compile resets trainable to True
• Expecting an error from accessing trainable
• Confusing trainable with layer existence

4. You want to unfreeze a previously frozen layer in your TensorFlow model. Which code snippet fixes the error below?

AttributeError: 'Sequential' object has no attribute 'unfreeze'

Current code:

model.unfreeze()
model.compile(optimizer='adam', loss='mse')

medium

A. Set trainable on each layer: for layer in model.layers: layer.trainable = True

B. Set model.trainable = False instead

C. Call model.unfreeze() method

D. Rebuild the model from scratch

Solution

1. Step 1: Understand the error cause

   The error says Sequential model has no unfreeze attribute, meaning calling model.unfreeze() is invalid.

2. Step 2: Correct way to unfreeze layers

   Each layer has a trainable attribute; set it to True for all layers to unfreeze.

3. Final Answer:

   Set trainable on each layer: for layer in model.layers: layer.trainable = True -> Option A

4. Quick Check:

   Unfreeze layers by setting layer.trainable True [OK]

Hint: Set trainable on layers, not on model object [OK]

Common Mistakes:

• Trying to call model.unfreeze()
• Using non-existent unfreeze() method
• Rebuilding model unnecessarily

5. You have a pretrained model and want to freeze all layers except the last two for fine-tuning. Which code correctly achieves this before compiling?

hard

A. for layer in model.layers: layer.trainable = True

B. for layer in model.layers[:-2]: layer.trainable = False
for layer in model.layers[-2:]: layer.trainable = True

C. model.trainable = False
model.layers[-2:].trainable = True

D. model.freeze_layers(except_last=2)

Solution

1. Step 1: Freeze all layers except last two

   Use slicing to set trainable = False for all layers except last two.

2. Step 2: Unfreeze last two layers

   Set trainable = True for the last two layers to allow training.

3. Final Answer:

   for layer in model.layers[:-2]: layer.trainable = False for layer in model.layers[-2:]: layer.trainable = True -> Option B

4. Quick Check:

   Freeze all but last two layers [OK]

Hint: Freeze with slicing; unfreeze last layers explicitly [OK]

Common Mistakes:

• Trying to set trainable on model object
• Using non-existent freeze_layers method
• Setting all layers trainable True