Optimizers (SGD, Adam, RMSprop) in TensorFlow

Introduction

Optimizers help a machine learning model learn by adjusting its settings to make better predictions.

When training a model to recognize images or sounds.

When you want your model to improve step-by-step during training.

When you need to choose how the model updates itself to reduce mistakes.

When comparing different ways to make your model learn faster or better.

When tuning your model to get the best accuracy on new data.

Syntax

TensorFlow

optimizer = tf.keras.optimizers.SGD(learning_rate=0.01, momentum=0.0)
optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)
optimizer = tf.keras.optimizers.RMSprop(learning_rate=0.001, rho=0.9)

You create an optimizer by calling its class with settings like learning rate.

Learning rate controls how big each update step is during training.

Examples

Simple SGD optimizer with a learning rate of 0.01.

TensorFlow

optimizer = tf.keras.optimizers.SGD(learning_rate=0.01)

Adam optimizer, good for many problems, with learning rate 0.001.

TensorFlow

optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)

RMSprop optimizer with decay rate rho set to 0.9.

TensorFlow

optimizer = tf.keras.optimizers.RMSprop(learning_rate=0.001, rho=0.9)

Sample Model

This code builds a small model and trains it with the Adam optimizer. It prints the loss after each training round to show learning progress.

TensorFlow

import tensorflow as tf

# Create a simple model
model = tf.keras.Sequential([
    tf.keras.layers.Dense(10, activation='relu', input_shape=(5,)),
    tf.keras.layers.Dense(1)
])

# Choose optimizer: SGD, Adam, or RMSprop
optimizer = tf.keras.optimizers.Adam(learning_rate=0.01)

# Compile model with optimizer and loss
model.compile(optimizer=optimizer, loss='mse')

# Create some dummy data
import numpy as np
x_train = np.random.random((100, 5))
y_train = np.random.random((100, 1))

# Train the model
history = model.fit(x_train, y_train, epochs=3, verbose=0)

# Print loss values after each epoch
for i, loss in enumerate(history.history['loss'], 1):
    print(f"Epoch {i}: loss = {loss:.4f}")

Important Notes

SGD is simple and works well for many tasks but can be slow to learn.

Adam adapts learning rates for each parameter, often leading to faster training.

RMSprop is good for problems with noisy or changing data.

Summary

Optimizers control how a model learns by updating its settings.

SGD, Adam, and RMSprop are popular optimizers with different strengths.

Choosing the right optimizer helps your model learn better and faster.

Practice

1. Which optimizer in TensorFlow uses momentum to accelerate gradient descent and reduce oscillations?

easy

A. SGD with momentum

B. Adam

C. RMSprop

D. Adagrad

Solution

1. Step 1: Understand momentum in optimizers

   Momentum helps speed up SGD by accumulating past gradients to smooth updates.

2. Step 2: Identify optimizer using momentum

   SGD with momentum explicitly uses this technique, unlike Adam or RMSprop which use adaptive learning rates.

3. Final Answer:

   SGD with momentum -> Option A

4. Quick Check:

   Momentum = SGD with momentum [OK]

Hint: Momentum is a feature of SGD, not Adam or RMSprop [OK]

Common Mistakes:

• Confusing Adam's adaptive learning with momentum
• Thinking RMSprop uses momentum
• Mixing up Adagrad with momentum

2. Which of the following is the correct way to create an Adam optimizer in TensorFlow with a learning rate of 0.001?

easy

A. tf.optimizers.Adam(lr=0.001)

B. tf.AdamOptimizer(0.001)

C. tf.optimizers.Adam(learning_rate=0.001)

D. tf.optimizers.AdamOptimizer(learning_rate=0.001)

Solution

1. Step 1: Recall TensorFlow 2.x optimizer syntax

   In TensorFlow 2.x, optimizers are created via tf.optimizers.OptimizerName with named parameters.

2. Step 2: Check correct Adam optimizer syntax

   The correct call is tf.optimizers.Adam(learning_rate=0.001). Other options use outdated or incorrect names.

3. Final Answer:

   tf.optimizers.Adam(learning_rate=0.001) -> Option C

4. Quick Check:

   Correct syntax = tf.optimizers.Adam(learning_rate=0.001) [OK]

Hint: Use tf.optimizers.Adam with named learning_rate [OK]

Common Mistakes:

• Using old tf.AdamOptimizer from TF1.x
• Passing learning rate as positional argument
• Using non-existent tf.optimizers.AdamOptimizer

3. What will be the output loss value after one training step using RMSprop optimizer with learning rate 0.01 on a simple linear model trained on data x=[1,2], y=[2,4]? Assume initial weights are zero and mean squared error loss.

medium

A. 0.5

B. 9.5

C. 1.0

D. 4.0

Solution

1. Step 1: Calculate initial prediction and loss

   Initial weights zero means prediction is 0 for inputs. Loss = mean squared error = mean([4,16]) = 10.

2. Step 2: Perform one RMSprop update step

   RMSprop scales update by rms of gradient (first step rms ≈ 0.32*|g|). Gradients ≈[-10,-6] for [w,b], updates ≈[+0.032,+0.032]. New preds ≈[0.063,0.095], new loss ≈9.5.

3. Final Answer:

   9.5 -> Option B

4. Quick Check:

   Loss after step ≈ 9.5 [OK]

Hint: RMSprop first step small due to scaling, loss ~9.5 [OK]

Common Mistakes:

• Expecting sharp loss drop after one step
• Confusing learning rate effect
• Ignoring initial zero weights impact

4. You wrote this code to use Adam optimizer but get an error:

optimizer = tf.optimizers.Adam(lr=0.01)
model.compile(optimizer=optimizer, loss='mse')

What is the likely cause of the error?

medium

A. Model.compile does not accept optimizer objects

B. Adam optimizer does not accept float arguments

C. Loss function 'mse' is invalid

D. Learning rate must be named as learning_rate=0.01

Solution

1. Step 1: Check Adam optimizer argument requirements

   TF2.x Adam expects keyword 'learning_rate=', not TF1.x-style 'lr='.

2. Step 2: Identify error cause in code

   Using lr=0.01 causes TypeError (unexpected keyword). Correct: tf.optimizers.Adam(learning_rate=0.01).

3. Final Answer:

   Learning rate must be named as learning_rate=0.01 -> Option D

4. Quick Check:

   Named argument needed [OK]

Hint: Always name learning_rate in Adam optimizer [OK]

Common Mistakes:

• Using 'lr=0.01' keyword from TF1.x
• Assuming 'mse' is invalid loss
• Thinking optimizer object can't be passed

5. You want to train a model on noisy data that changes over time. Which optimizer is best suited to adapt learning rates per parameter and handle this noise effectively?

hard

A. Adam

B. Gradient Descent with fixed learning rate

C. RMSprop

D. SGD without momentum

Solution

1. Step 1: Understand optimizer strengths for noisy data

   Adam adapts learning rates per parameter and combines momentum and RMSprop ideas, handling noise well.

2. Step 2: Compare with other optimizers

   SGD without momentum and fixed learning rate struggle with noise. RMSprop adapts rates but Adam adds momentum for better stability.

3. Final Answer:

   Adam -> Option A

4. Quick Check:

   Best for noisy data = Adam [OK]

Hint: Adam adapts learning rates and handles noise best [OK]

Common Mistakes:

• Choosing plain SGD for noisy data
• Confusing RMSprop with Adam's momentum
• Ignoring adaptive learning rate benefits