Callbacks (EarlyStopping, ModelCheckpoint) in TensorFlow

Introduction

Callbacks help your model learn better by stopping early when it stops improving and saving the best version automatically.

When you want to stop training if the model stops getting better to save time and avoid overfitting.

When you want to save the best model during training to use it later without retraining.

When training takes a long time and you want to keep the best checkpoint in case of interruptions.

When you want to monitor validation performance and act on it automatically during training.

Syntax

TensorFlow

from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint

early_stop = EarlyStopping(monitor='val_loss', patience=3, verbose=1)
model_checkpoint = ModelCheckpoint('best_model.h5', monitor='val_loss', save_best_only=True, verbose=1)

monitor: The metric to watch, like 'val_loss' or 'val_accuracy'.

patience: How many bad epochs to wait before stopping.

Examples

Stops training if validation accuracy does not improve for 5 epochs.

TensorFlow

early_stop = EarlyStopping(monitor='val_accuracy', patience=5, verbose=1)

Saves the model weights only when validation loss improves.

TensorFlow

model_checkpoint = ModelCheckpoint('best_weights.h5', monitor='val_loss', save_best_only=True, save_weights_only=True, verbose=1)

Stops early and restores the best weights found during training.

TensorFlow

early_stop = EarlyStopping(monitor='val_loss', patience=2, restore_best_weights=True)

Sample Model

This code trains a simple neural network on random data. It uses EarlyStopping to stop if validation loss doesn't improve for 3 epochs and ModelCheckpoint to save the best model. After training, it prints validation loss and accuracy.

TensorFlow

import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint

# Create simple model
model = Sequential([
    Dense(16, activation='relu', input_shape=(10,)),
    Dense(1, activation='sigmoid')
])

model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

# Generate dummy data
import numpy as np
x_train = np.random.random((100, 10))
y_train = np.random.randint(2, size=(100, 1))
x_val = np.random.random((20, 10))
y_val = np.random.randint(2, size=(20, 1))

# Setup callbacks
early_stop = EarlyStopping(monitor='val_loss', patience=3, verbose=1, restore_best_weights=True)
model_checkpoint = ModelCheckpoint('best_model.h5', monitor='val_loss', save_best_only=True, verbose=1)

# Train model with callbacks
history = model.fit(
    x_train, y_train,
    epochs=20,
    batch_size=10,
    validation_data=(x_val, y_val),
    callbacks=[early_stop, model_checkpoint],
    verbose=2
)

# Evaluate model
loss, accuracy = model.evaluate(x_val, y_val, verbose=0)
print(f'Validation loss: {loss:.4f}')
print(f'Validation accuracy: {accuracy:.4f}')

Important Notes

EarlyStopping helps avoid wasting time training when the model stops improving.

ModelCheckpoint saves your best model automatically, so you don't lose progress.

Use restore_best_weights=True in EarlyStopping to keep the best model after stopping.

Summary

Callbacks like EarlyStopping and ModelCheckpoint improve training by saving time and best models.

EarlyStopping stops training when no improvement happens for a set patience.

ModelCheckpoint saves the best model during training automatically.

Practice

1. What is the main purpose of the EarlyStopping callback in TensorFlow training?

easy

A. To increase the learning rate during training

B. To save the model weights after every epoch

C. To stop training when the model stops improving to save time

D. To shuffle the training data before each epoch

Solution

1. Step 1: Understand EarlyStopping's role

   EarlyStopping monitors a metric like validation loss and stops training if no improvement occurs for a set number of epochs.

2. Step 2: Compare options with EarlyStopping behavior

   Only To stop training when the model stops improving to save time describes stopping training to save time when no improvement happens.

3. Final Answer:

   To stop training when the model stops improving to save time -> Option C

4. Quick Check:

   EarlyStopping stops training early = C [OK]

Hint: EarlyStopping stops training early to save time [OK]

Common Mistakes:

• Confusing EarlyStopping with saving models
• Thinking EarlyStopping changes learning rate
• Assuming EarlyStopping shuffles data

2. Which of the following is the correct way to create a ModelCheckpoint callback that saves only the best model based on validation accuracy?

easy

A. tf.keras.callbacks.ModelCheckpoint('best_model.h5', save_best_only=False, monitor='accuracy')

B. tf.keras.callbacks.ModelCheckpoint('best_model.h5', save_best_only=True, monitor='val_accuracy')

C. tf.keras.callbacks.ModelCheckpoint('best_model.h5', save_weights_only=True, monitor='val_loss')

D. tf.keras.callbacks.ModelCheckpoint('best_model.h5', save_best_only=True, monitor='loss')

Solution

1. Step 1: Identify correct parameters for ModelCheckpoint

   To save only the best model, save_best_only=True is needed, and to monitor validation accuracy, monitor='val_accuracy' is correct.

2. Step 2: Check options for matching parameters

   tf.keras.callbacks.ModelCheckpoint('best_model.h5', save_best_only=True, monitor='val_accuracy') matches these requirements exactly.

3. Final Answer:

   tf.keras.callbacks.ModelCheckpoint('best_model.h5', save_best_only=True, monitor='val_accuracy') -> Option B

4. Quick Check:

   Best model saved by val_accuracy = A [OK]

Hint: Use save_best_only=True and monitor='val_accuracy' [OK]

Common Mistakes:

• Using monitor='accuracy' instead of 'val_accuracy'
• Setting save_best_only=False by mistake
• Confusing save_weights_only with saving full model

3. Consider this code snippet using EarlyStopping:

callback = tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=2)
model.fit(x_train, y_train, epochs=10, validation_data=(x_val, y_val), callbacks=[callback])

If the validation loss stops improving after epoch 4, at which epoch will training stop?

medium

A. Epoch 4

B. Epoch 10

C. Epoch 5

D. Epoch 7

Solution

1. Step 1: Understand patience parameter in EarlyStopping

   Patience=2 means training continues 2 more epochs after last improvement before stopping.

2. Step 2: Calculate stopping epoch

   If last improvement is at epoch 4, training continues epochs 5 and 6, then stops before epoch 7 starts, so training stops at epoch 7.

3. Final Answer:

   Epoch 7 -> Option D

4. Quick Check:

   Patience 2 means stop 2 epochs after no improvement = B [OK]

Hint: Training stops after patience epochs without improvement [OK]

Common Mistakes:

• Stopping immediately at last improvement epoch
• Stopping one epoch too early or too late
• Confusing patience with number of total epochs

4. You wrote this code but the model never stops early even when validation loss stops improving:

callback = tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=3)
model.fit(x_train, y_train, epochs=20, validation_data=(x_val, y_val), callbacks=[callback])

What is the most likely reason training does not stop early?

medium

A. The validation data is not passed correctly, so val_loss is not computed

B. Patience is too low to allow stopping

C. EarlyStopping requires save_best_only=True to work

D. The model.fit call is missing the callbacks argument

Solution

1. Step 1: Check if validation data is correctly passed

   EarlyStopping monitors validation metrics, so if validation data is missing or incorrect, val_loss won't update and stopping won't trigger.

2. Step 2: Evaluate other options

   Patience=3 is reasonable, save_best_only is unrelated to EarlyStopping, and callbacks argument is present.

3. Final Answer:

   The validation data is not passed correctly, so val_loss is not computed -> Option A

4. Quick Check:

   EarlyStopping needs valid val_loss metric = D [OK]

Hint: EarlyStopping needs valid validation data to monitor val_loss [OK]

Common Mistakes:

• Confusing ModelCheckpoint's save_best_only with EarlyStopping
• Ignoring validation_data argument
• Setting patience too high and expecting early stop

5. You want to train a model and save the best weights based on validation accuracy, but also stop training early if validation accuracy does not improve for 4 epochs. Which callback setup is correct?

hard

A. [tf.keras.callbacks.EarlyStopping(monitor='val_accuracy', patience=4), tf.keras.callbacks.ModelCheckpoint('best.h5', save_best_only=True, monitor='val_accuracy')]

B. [tf.keras.callbacks.EarlyStopping(monitor='accuracy', patience=4), tf.keras.callbacks.ModelCheckpoint('best.h5', save_best_only=False, monitor='val_accuracy')]

C. [tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=4), tf.keras.callbacks.ModelCheckpoint('best.h5', save_best_only=True, monitor='loss')]

D. [tf.keras.callbacks.EarlyStopping(monitor='val_accuracy', patience=2), tf.keras.callbacks.ModelCheckpoint('best.h5', save_best_only=True, monitor='val_accuracy')]

Solution

1. Step 1: Match EarlyStopping parameters to requirement

   We want to stop if validation accuracy does not improve for 4 epochs, so monitor='val_accuracy' and patience=4 are correct.

2. Step 2: Match ModelCheckpoint parameters

   We want to save best weights based on validation accuracy, so save_best_only=True and monitor='val_accuracy' are needed.

3. Step 3: Check options for both callbacks

   Only [tf.keras.callbacks.EarlyStopping(monitor='val_accuracy', patience=4), tf.keras.callbacks.ModelCheckpoint('best.h5', save_best_only=True, monitor='val_accuracy')] has both callbacks correctly configured.

4. Final Answer:

   [tf.keras.callbacks.EarlyStopping(monitor='val_accuracy', patience=4), tf.keras.callbacks.ModelCheckpoint('best.h5', save_best_only=True, monitor='val_accuracy')] -> Option A

5. Quick Check:

   EarlyStopping and ModelCheckpoint monitor val_accuracy correctly = A [OK]

Hint: Match monitor and patience for both callbacks [OK]

Common Mistakes:

• Using 'accuracy' instead of 'val_accuracy' for validation monitoring
• Setting save_best_only=False when saving best model
• Mismatching patience with requirement