Learning rate scheduling in TensorFlow - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

What is the main purpose of learning rate scheduling in training neural networks?

Why do we use learning rate scheduling when training a neural network?

A To gradually reduce the learning rate to help the model converge better and avoid overshooting minima.

B To increase the learning rate over time to speed up training indefinitely.

C To keep the learning rate constant throughout training for stable updates.

D To randomly change the learning rate each epoch to add noise and prevent overfitting.

💡 Hint

Think about how changing the learning rate affects the model's ability to find the best solution.

Learning rate scheduling helps by starting with a higher learning rate for faster learning, then reducing it to fine-tune the model and avoid jumping over the best solution.

❓ Predict Output

intermediate

Output of learning rate after 5 epochs with ExponentialDecay

Given this TensorFlow learning rate schedule, what is the learning rate at epoch 5?

TensorFlow

import tensorflow as tf
initial_lr = 0.1
lr_schedule = tf.keras.optimizers.schedules.ExponentialDecay(
    initial_learning_rate=initial_lr,
    decay_steps=2,
    decay_rate=0.5,
    staircase=True
)
learning_rate_epoch_5 = lr_schedule(5).numpy()
print(round(learning_rate_epoch_5, 4))

A 0.1

B 0.0125

C 0.05

D 0.025

💡 Hint

Remember that with staircase=True, the learning rate changes only at multiples of decay_steps.

With decay_steps=2 and staircase=True, the learning rate decays every 2 steps. At step 5, decay count is floor(5/2)=2, so lr = 0.1 * 0.5^2 = 0.025.

❓ Model Choice

advanced

Which learning rate schedule is best for training a model that needs slow fine-tuning after initial fast learning?

You want a learning rate schedule that starts high and then slowly decreases to fine-tune the model. Which schedule fits best?

A Cosine annealing schedule that gradually reduces learning rate following a cosine curve.

B Constant learning rate with no changes during training.

C Step decay schedule that drops learning rate sharply at fixed intervals.

D Random learning rate changes each epoch.

💡 Hint

Think about smooth gradual decrease versus sudden drops.

Cosine annealing smoothly reduces the learning rate, allowing initial fast learning and slow fine-tuning later, unlike step decay which drops sharply.

❓ Hyperparameter

advanced

Effect of increasing decay_steps in ExponentialDecay schedule

In TensorFlow's ExponentialDecay schedule, what happens if you increase the decay_steps parameter while keeping others constant?

A The learning rate decays faster over time.

B The learning rate decays more slowly over time.

C The learning rate stays constant and does not decay.

D The learning rate increases over time.

💡 Hint

Think about how often the decay happens with bigger decay_steps.

Increasing decay_steps means the decay happens less frequently, so the learning rate decreases more slowly.

❓ Metrics

expert

How does learning rate scheduling affect training loss and accuracy curves?

When using a learning rate schedule that reduces the learning rate over epochs, what typical pattern do you expect to see in training loss and accuracy graphs?

A Training loss increases and accuracy decreases as learning rate lowers.

B Training loss and accuracy fluctuate wildly with no clear trend.

C Training loss decreases smoothly and accuracy increases steadily, with smaller improvements as learning rate lowers.

D Training loss and accuracy remain constant throughout training.

💡 Hint

Consider how smaller learning rates affect model updates and convergence.

Lower learning rates help the model make finer adjustments, so loss decreases smoothly and accuracy improves steadily but more slowly.