Optimizers (SGD, Adam, RMSprop) in TensorFlow

Introduction

Optimizers help a machine learning model learn by adjusting its settings to make better predictions.

When training a model to recognize images or sounds.

When you want your model to improve step-by-step during training.

When you need to choose how the model updates itself to reduce mistakes.

When comparing different ways to make your model learn faster or better.

When tuning your model to get the best accuracy on new data.

Syntax

TensorFlow

optimizer = tf.keras.optimizers.SGD(learning_rate=0.01, momentum=0.0)
optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)
optimizer = tf.keras.optimizers.RMSprop(learning_rate=0.001, rho=0.9)

You create an optimizer by calling its class with settings like learning rate.

Learning rate controls how big each update step is during training.

Examples

Simple SGD optimizer with a learning rate of 0.01.

TensorFlow

optimizer = tf.keras.optimizers.SGD(learning_rate=0.01)

Adam optimizer, good for many problems, with learning rate 0.001.

TensorFlow

optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)

RMSprop optimizer with decay rate rho set to 0.9.

TensorFlow

optimizer = tf.keras.optimizers.RMSprop(learning_rate=0.001, rho=0.9)

Sample Model

This code builds a small model and trains it with the Adam optimizer. It prints the loss after each training round to show learning progress.

TensorFlow

import tensorflow as tf

# Create a simple model
model = tf.keras.Sequential([
    tf.keras.layers.Dense(10, activation='relu', input_shape=(5,)),
    tf.keras.layers.Dense(1)
])

# Choose optimizer: SGD, Adam, or RMSprop
optimizer = tf.keras.optimizers.Adam(learning_rate=0.01)

# Compile model with optimizer and loss
model.compile(optimizer=optimizer, loss='mse')

# Create some dummy data
import numpy as np
x_train = np.random.random((100, 5))
y_train = np.random.random((100, 1))

# Train the model
history = model.fit(x_train, y_train, epochs=3, verbose=0)

# Print loss values after each epoch
for i, loss in enumerate(history.history['loss'], 1):
    print(f"Epoch {i}: loss = {loss:.4f}")

Important Notes

SGD is simple and works well for many tasks but can be slow to learn.

Adam adapts learning rates for each parameter, often leading to faster training.

RMSprop is good for problems with noisy or changing data.

Summary

Optimizers control how a model learns by updating its settings.

SGD, Adam, and RMSprop are popular optimizers with different strengths.

Choosing the right optimizer helps your model learn better and faster.