Learning rate scheduling in TensorFlow

Introduction

Learning rate scheduling helps the model learn better by changing the speed of learning during training. It starts fast and slows down to improve accuracy.

When training a neural network and you want to improve accuracy over time.

When the model stops improving and you want to try slowing down learning to fine-tune.

When training takes a long time and you want to avoid overshooting the best solution.

When you want to avoid the model jumping around too much in the learning process.

Syntax

TensorFlow

import tensorflow as tf

# Example: create a learning rate schedule
schedule = tf.keras.optimizers.schedules.ExponentialDecay(
    initial_learning_rate=0.1,
    decay_steps=10000,
    decay_rate=0.96,
    staircase=True
)

# Use schedule in optimizer
optimizer = tf.keras.optimizers.SGD(learning_rate=schedule)

The learning rate schedule changes the learning rate during training automatically.

Common schedules include exponential decay, step decay, and cosine decay.

Examples

This schedule reduces the learning rate by 10% every 1000 steps smoothly.

TensorFlow

schedule = tf.keras.optimizers.schedules.ExponentialDecay(
    initial_learning_rate=0.01,
    decay_steps=1000,
    decay_rate=0.9
)

This schedule uses fixed learning rates that change at step 1000 and 2000.

TensorFlow

schedule = tf.keras.optimizers.schedules.PiecewiseConstantDecay(
    boundaries=[1000, 2000],
    values=[0.1, 0.01, 0.001]
)

This schedule reduces the learning rate following a cosine curve over 10000 steps.

TensorFlow

schedule = tf.keras.optimizers.schedules.CosineDecay(
    initial_learning_rate=0.1,
    decay_steps=10000
)

Sample Model

This program trains a simple model to learn y=2x+1 using a learning rate that halves every 2 steps. It shows how the learning rate changes and the prediction after training.

TensorFlow

import tensorflow as tf
import numpy as np

# Create simple data: y = 2x + 1
x_train = np.array([[1.0], [2.0], [3.0], [4.0]], dtype=np.float32)
y_train = np.array([[3.0], [5.0], [7.0], [9.0]], dtype=np.float32)

# Define a simple linear model
model = tf.keras.Sequential([
    tf.keras.layers.Dense(1, input_shape=(1,))
])

# Define learning rate schedule
lr_schedule = tf.keras.optimizers.schedules.ExponentialDecay(
    initial_learning_rate=0.1,
    decay_steps=2,
    decay_rate=0.5,
    staircase=True
)

# Use SGD optimizer with learning rate schedule
optimizer = tf.keras.optimizers.SGD(learning_rate=lr_schedule)

# Compile model
model.compile(optimizer=optimizer, loss='mse')

# Train model
history = model.fit(x_train, y_train, epochs=5, verbose=2)

# Predict
predictions = model.predict(np.array([[5.0]]))

print(f"Prediction for input 5.0: {predictions[0][0]:.4f}")

Important Notes

Learning rate schedules help avoid getting stuck or jumping too much during training.

Choosing the right schedule and parameters can improve model results significantly.

Summary

Learning rate scheduling changes the learning speed during training.

It helps the model learn better and reach higher accuracy.

TensorFlow offers many built-in schedules like exponential and cosine decay.

Practice

1. What is the main purpose of learning rate scheduling in TensorFlow training?

easy

A. To adjust the learning speed during training for better results

B. To increase the batch size automatically

C. To change the model architecture dynamically

D. To stop training early when accuracy is high

Solution

1. Step 1: Understand learning rate scheduling concept

   Learning rate scheduling changes how fast the model learns by adjusting the learning rate during training.

2. Step 2: Identify the main goal

   The goal is to help the model learn better and reach higher accuracy by controlling the learning speed.

3. Final Answer:

   To adjust the learning speed during training for better results -> Option A

4. Quick Check:

   Learning rate scheduling = adjust learning speed [OK]

Hint: Learning rate scheduling controls training speed [OK]

Common Mistakes:

• Confusing learning rate scheduling with batch size changes
• Thinking it changes model structure
• Assuming it stops training early

2. Which of the following is the correct way to create an exponential decay learning rate schedule in TensorFlow?

easy

A. tf.keras.optimizers.schedules.ExponentialDecay(initial_learning_rate=0.1, decay_steps=1000, decay_rate=0.96)

B. tf.keras.optimizers.ExponentialDecay(learning_rate=0.1, steps=1000, rate=0.96)

C. tf.keras.schedules.ExponentialDecay(learning_rate=0.1, decay_steps=1000, decay_rate=0.96)

D. tf.optimizers.schedules.ExponentialDecay(initial_lr=0.1, decay_steps=1000, decay_rate=0.96)

Solution

1. Step 1: Recall TensorFlow API for exponential decay

   The correct class is tf.keras.optimizers.schedules.ExponentialDecay with parameters initial_learning_rate, decay_steps, and decay_rate.

2. Step 2: Match parameters and class name

   tf.keras.optimizers.schedules.ExponentialDecay(initial_learning_rate=0.1, decay_steps=1000, decay_rate=0.96) matches the correct class and parameter names exactly.

3. Final Answer:

   tf.keras.optimizers.schedules.ExponentialDecay(initial_learning_rate=0.1, decay_steps=1000, decay_rate=0.96) -> Option A

4. Quick Check:

   Correct class and parameters = tf.keras.optimizers.schedules.ExponentialDecay(initial_learning_rate=0.1, decay_steps=1000, decay_rate=0.96) [OK]

Hint: Use tf.keras.optimizers.schedules for learning rate schedules [OK]

Common Mistakes:

• Using wrong module like tf.keras.optimizers directly
• Incorrect parameter names like learning_rate instead of initial_learning_rate
• Misspelling class or method names

3. What will be the learning rate after 2000 steps if you use this schedule?

schedule = tf.keras.optimizers.schedules.ExponentialDecay(
    initial_learning_rate=0.1,
    decay_steps=1000,
    decay_rate=0.5)
learning_rate = schedule(2000).numpy()

medium

A. 0.05

B. 0.01

C. 0.1

D. 0.025

Solution

1. Step 1: Understand exponential decay formula

   Learning rate = initial_learning_rate * decay_rate^(step / decay_steps). Here, step=2000, decay_steps=1000, decay_rate=0.5.

2. Step 2: Calculate learning rate at step 2000

   Calculate exponent: 2000 / 1000 = 2.
   Learning rate = 0.1 * (0.5)^2 = 0.1 * 0.25 = 0.025.

3. Final Answer:

   0.025 -> Option D

4. Quick Check:

   0.1 * 0.5^2 = 0.025 [OK]

Hint: Apply formula: lr = initial_lr * decay_rate^(step/decay_steps) [OK]

Common Mistakes:

• Forgetting to divide step by decay_steps
• Using decay_rate power step instead of step/decay_steps
• Mixing up decay_rate and initial_learning_rate

4. Identify the error in this TensorFlow learning rate schedule code:

schedule = tf.keras.optimizers.schedules.ExponentialDecay(
    initial_learning_rate=0.01,
    decay_steps=500,
    decay_rate=0.9)
optimizer = tf.keras.optimizers.Adam(learning_rate=schedule())

medium

A. Adam optimizer does not accept learning rate schedules

B. schedule() is called without step argument, causing an error

C. Decay rate 0.9 is too high and causes slow decay

D. initial_learning_rate should be set in optimizer, not schedule

Solution

1. Step 1: Check how schedule is used

   ExponentialDecay expects a step argument to compute learning rate. Calling schedule() without arguments causes an error.

2. Step 2: Verify optimizer learning rate parameter

   Optimizer accepts schedule object directly, not schedule() call. Passing schedule() calls the function immediately without step.

3. Final Answer:

   schedule() is called without step argument, causing an error -> Option B

4. Quick Check:

   Call schedule without step causes error [OK]

Hint: Pass schedule object, don't call it with () in optimizer [OK]

Common Mistakes:

• Calling schedule() instead of passing schedule object
• Thinking decay_rate value causes error
• Believing Adam can't use schedules

5. You want to train a model where the learning rate starts at 0.1, halves every 1000 steps, but never goes below 0.01. Which TensorFlow schedule setup achieves this?

hard

A. Use CosineDecay with initial_learning_rate=0.1 and alpha=0.01 to set minimum learning rate

B. Use ExponentialDecay with initial_learning_rate=0.1, decay_steps=1000, decay_rate=0.5 and then apply tf.keras.optimizers.schedules.PiecewiseConstantDecay with boundaries=[1000] and values=[0.1, 0.01]

C. Use ExponentialDecay with initial_learning_rate=0.1, decay_steps=1000, decay_rate=0.5 and then use tf.keras.optimizers.schedules.LearningRateSchedule subclass to clip minimum at 0.01

D. Use ExponentialDecay with initial_learning_rate=0.1, decay_steps=1000, decay_rate=0.5 and wrap it with tf.keras.optimizers.schedules.LearningRateSchedule to clip minimum at 0.01

Solution

1. Step 1: Understand the requirement

   Learning rate halves every 1000 steps: ExponentialDecay(initial_learning_rate=0.1, decay_steps=1000, decay_rate=0.5), but never below 0.01 (needs clipping).

2. Step 2: Evaluate schedules

   ExponentialDecay provides exact halving at multiples of decay_steps but lacks built-in minimum. Custom clipping needed.

3. Step 3: Identify correct approach

   Use ExponentialDecay with initial_learning_rate=0.1, decay_steps=1000, decay_rate=0.5 and then use tf.keras.optimizers.schedules.LearningRateSchedule subclass to clip minimum at 0.01 subclasses LearningRateSchedule to wrap ExponentialDecay and clip with tf.maximum(..., 0.01).

4. Final Answer:

   Use ExponentialDecay with initial_learning_rate=0.1, decay_steps=1000, decay_rate=0.5 and then use tf.keras.optimizers.schedules.LearningRateSchedule subclass to clip minimum at 0.01 -> Option C

5. Quick Check:

   ExponentialDecay + subclass clip = halving with floor [OK]

Hint: Subclass LearningRateSchedule to clip ExponentialDecay minimum [OK]

Common Mistakes:

• Using CosineDecay which does smooth decay, not discrete halving every 1000 steps
• Using PiecewiseConstantDecay incorrectly for continuous exponential decay
• Attempting to wrap without proper subclassing