Custom callbacks in TensorFlow - Model Metrics & Evaluation

Metrics & Evaluation - Custom callbacks

Which metric matters for Custom Callbacks and WHY

Custom callbacks let you track and control training steps like saving models or stopping early. The key metrics to watch depend on your goal. For example, if you want to stop training when validation loss stops improving, validation loss is the metric to monitor. If you save the best model, you track validation accuracy or validation loss. Custom callbacks help you react to these metrics during training.

Confusion Matrix or Equivalent Visualization

Custom callbacks do not directly produce confusion matrices, but you can create callbacks to compute and display confusion matrices at the end of each epoch. For example, after predictions on validation data, you can build a confusion matrix like this:

Confusion Matrix:
          Predicted
          P     N
    Actual P  TP    FN
           N  FP    TN

Where:
TP = True Positives
FP = False Positives
TN = True Negatives
FN = False Negatives

A callback can compute these counts and print or log them during training.

Precision vs Recall Tradeoff with Examples

Custom callbacks can help monitor precision and recall during training. For example, if you want to improve recall (catch more positives), you can stop training early when recall stops improving. Or if you want to avoid false alarms, you can focus on precision.

Example: In a cancer detection model, recall is critical because missing a cancer case is dangerous. A custom callback can stop training if recall drops, even if accuracy is high.

Custom callbacks let you choose which metric to prioritize and act on it during training.

What "Good" vs "Bad" Metric Values Look Like for Custom Callbacks

Good metric values depend on your task and what your callback monitors:

• Good: Validation loss steadily decreases, validation accuracy increases, and your callback saves the best model at the right time.
• Bad: Validation loss bounces up and down, accuracy does not improve, or your callback stops training too early or too late.

Custom callbacks help you automate these decisions based on metric trends.

Common Metrics Pitfalls with Custom Callbacks

• Accuracy paradox: High accuracy can be misleading if classes are imbalanced. Callbacks should monitor metrics like precision or recall instead.
• Data leakage: If validation data leaks into training, callbacks may save models that look good but fail in real use.
• Overfitting indicators: If validation loss increases while training loss decreases, callbacks can stop training early to avoid overfitting.
• Improper metric monitoring: Callbacks must monitor the correct metric (e.g., 'val_loss' vs 'loss') to work as intended.

Self Check: Your model has 98% accuracy but 12% recall on fraud. Is it good?

No, this model is not good for fraud detection. Even though accuracy is high, recall is very low. This means the model misses most fraud cases, which is dangerous. For fraud detection, recall is critical because missing fraud is worse than false alarms. You should improve recall, possibly by adjusting the model or using a custom callback to monitor and stop training based on recall.

Key Result

Custom callbacks help monitor and act on key metrics like validation loss, accuracy, precision, and recall during training to improve model performance.

Practice

1. What is the main purpose of creating a custom callback in TensorFlow?

easy

A. To run your own code at specific points during model training

B. To change the model architecture automatically

C. To increase the dataset size during training

D. To convert the model to a different format

Solution

1. Step 1: Understand what callbacks do

   Callbacks let you add custom actions during training, like logging or stopping early.

2. Step 2: Identify the role of custom callbacks

   Custom callbacks let you run your own code at points like start/end of epochs or batches.

3. Final Answer:

   To run your own code at specific points during model training -> Option A

4. Quick Check:

   Custom callbacks = run code during training [OK]

Hint: Callbacks run code during training steps [OK]

Common Mistakes:

• Thinking callbacks change model structure
• Confusing callbacks with data augmentation
• Assuming callbacks convert model formats

2. Which method should you override in a custom callback to execute code at the end of each training epoch?

easy

A. on_train_begin

B. on_epoch_end

C. on_batch_begin

D. on_train_end

Solution

1. Step 1: Recall callback method names

   Common callback methods include on_train_begin, on_epoch_end, on_batch_begin, on_train_end.

2. Step 2: Match method to event

   on_epoch_end runs code after each epoch finishes, which is what the question asks.

3. Final Answer:

   on_epoch_end -> Option B

4. Quick Check:

   End of epoch method = on_epoch_end [OK]

Hint: End of epoch callback method is on_epoch_end [OK]

Common Mistakes:

• Using on_train_begin for epoch end
• Confusing batch and epoch methods
• Overriding on_train_end instead

3. What will be printed after training if this custom callback is used?

import tensorflow as tf
class MyCallback(tf.keras.callbacks.Callback):
    def on_epoch_end(self, epoch, logs=None):
        print(f"Epoch {epoch+1} ended with loss {logs['loss']:.2f}")

model = tf.keras.Sequential([tf.keras.layers.Dense(1)])
model.compile(optimizer='sgd', loss='mse')
model.fit([[1.0]], [[2.0]], epochs=2, callbacks=[MyCallback()])

medium

A. Error because logs is None

B. Epoch 0 ended with loss X\nEpoch 1 ended with loss Y

C. No output printed during training

D. Epoch 1 ended with loss X\nEpoch 2 ended with loss Y (X and Y are numbers)

Solution

1. Step 1: Understand on_epoch_end behavior

   The method prints epoch number +1 and loss from logs after each epoch.

2. Step 2: Check training epochs and output

   Training runs 2 epochs, so prints for epoch 1 and 2 with loss values (numbers).

3. Final Answer:

   Epoch 1 ended with loss X\nEpoch 2 ended with loss Y (X and Y are numbers) -> Option D

4. Quick Check:

   Prints after each epoch with loss [OK]

Hint: Epoch count starts at 0, print adds 1 [OK]

Common Mistakes:

• Assuming epoch numbering starts at 1 internally
• Expecting no print output
• Thinking logs is None causing error

4. Identify the error in this custom callback code that tries to stop training when loss is below 0.1:

class StopCallback(tf.keras.callbacks.Callback):
    def on_epoch_end(self, epoch, logs=None):
        if logs['loss'] < 0.1:
            self.model.stop_training = True

model.fit(x_train, y_train, epochs=10, callbacks=[StopCallback()])

medium

A. on_epoch_end method must return a value

B. stop_training should be called as a method

C. logs dictionary might be None causing KeyError

D. Callback class must inherit from tf.keras.Model

Solution

1. Step 1: Check logs usage

   logs can be None if no metrics are passed, so logs['loss'] may cause KeyError.

2. Step 2: Confirm correct way to stop training

   Setting self.model.stop_training = True is correct; no method call needed.

3. Final Answer:

   logs dictionary might be None causing KeyError -> Option C

4. Quick Check:

   Check logs is not None before accessing [OK]

Hint: Always check logs is not None before using keys [OK]

Common Mistakes:

• Calling stop_training as a method
• Expecting on_epoch_end to return a value
• Inheriting from wrong base class

5. You want to save the model weights only when validation accuracy improves during training. Which custom callback method and logic should you implement?

hard

A. Override on_epoch_end, check logs['val_accuracy'], save weights if improved

B. Override on_train_begin, save weights before training starts

C. Override on_batch_end, save weights after every batch

D. Override on_train_end, save weights after training finishes

Solution

1. Step 1: Identify when to check validation accuracy

   Validation accuracy is updated at epoch end, so on_epoch_end is the right method.

2. Step 2: Implement logic to save weights only if val_accuracy improves

   Inside on_epoch_end, compare current val_accuracy with previous best and save if better.

3. Final Answer:

   Override on_epoch_end, check logs['val_accuracy'], save weights if improved -> Option A

4. Quick Check:

   Save on epoch end if val_accuracy improves [OK]

Hint: Use on_epoch_end to monitor validation metrics [OK]

Common Mistakes:

• Saving weights too often (every batch)
• Saving only before or after all training
• Not checking if accuracy improved before saving