Optimizers (SGD, Adam, RMSprop) in TensorFlow - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is the main role of an optimizer in machine learning?

An optimizer helps the model learn by adjusting its internal settings (weights) to reduce errors and improve predictions.

beginner

How does the SGD optimizer update model weights?

SGD (Stochastic Gradient Descent) updates weights by moving them a small step opposite to the error direction, using a fixed learning rate.

intermediate

What makes Adam optimizer different from SGD?

Adam combines ideas from momentum and RMSprop, adapting learning rates for each weight individually, which helps faster and more stable learning.

intermediate

Why is RMSprop useful for training neural networks?

RMSprop adjusts the learning rate for each weight based on recent gradients, helping the model learn well even when gradients vary a lot.

beginner

Which optimizer would you choose for a simple linear model and why?

SGD is often chosen for simple models because it is straightforward and effective when the learning rate is well tuned.

Which optimizer adapts the learning rate for each parameter individually?

A SGD

B Adam

C Batch Gradient Descent

D None of the above

Adam adjusts learning rates for each parameter based on past gradients, unlike SGD which uses a fixed rate.

What does SGD stand for?

A Stochastic Gradient Descent

B Simple Gradient Descent

C Sequential Gradient Descent

D Standard Gradient Descent

SGD means Stochastic Gradient Descent, which updates weights using small random batches.

Which optimizer uses a moving average of squared gradients to adjust learning rates?

A SGD

B Adam

C RMSprop

D Momentum

RMSprop uses a moving average of squared gradients to adapt learning rates for each parameter.

Why might Adam be preferred over SGD?

A It adapts learning rates and converges faster

B It is simpler to implement

C It requires no learning rate

D It uses less memory

Adam adapts learning rates for each parameter and often converges faster than SGD.

Which optimizer is best described as 'simple and effective with a fixed learning rate'?

A Adagrad

B RMSprop

C Adam

D SGD

SGD uses a fixed learning rate and is simple and effective for many problems.

Explain how the Adam optimizer works and why it might be better than SGD for some problems.

Think about how Adam changes learning rates for each weight and uses past gradients.

Describe the differences between SGD, RMSprop, and Adam optimizers in simple terms.

Focus on how each optimizer changes learning rates and uses past information.

Practice

1. Which optimizer in TensorFlow uses momentum to accelerate gradient descent and reduce oscillations?

easy

A. SGD with momentum

B. Adam

C. RMSprop

D. Adagrad

Solution

1. Step 1: Understand momentum in optimizers

   Momentum helps speed up SGD by accumulating past gradients to smooth updates.

2. Step 2: Identify optimizer using momentum

   SGD with momentum explicitly uses this technique, unlike Adam or RMSprop which use adaptive learning rates.

3. Final Answer:

   SGD with momentum -> Option A

4. Quick Check:

   Momentum = SGD with momentum [OK]

Hint: Momentum is a feature of SGD, not Adam or RMSprop [OK]

Common Mistakes:

• Confusing Adam's adaptive learning with momentum
• Thinking RMSprop uses momentum
• Mixing up Adagrad with momentum

2. Which of the following is the correct way to create an Adam optimizer in TensorFlow with a learning rate of 0.001?

easy

A. tf.optimizers.Adam(lr=0.001)

B. tf.AdamOptimizer(0.001)

C. tf.optimizers.Adam(learning_rate=0.001)

D. tf.optimizers.AdamOptimizer(learning_rate=0.001)

Solution

1. Step 1: Recall TensorFlow 2.x optimizer syntax

   In TensorFlow 2.x, optimizers are created via tf.optimizers.OptimizerName with named parameters.

2. Step 2: Check correct Adam optimizer syntax

   The correct call is tf.optimizers.Adam(learning_rate=0.001). Other options use outdated or incorrect names.

3. Final Answer:

   tf.optimizers.Adam(learning_rate=0.001) -> Option C

4. Quick Check:

   Correct syntax = tf.optimizers.Adam(learning_rate=0.001) [OK]

Hint: Use tf.optimizers.Adam with named learning_rate [OK]

Common Mistakes:

• Using old tf.AdamOptimizer from TF1.x
• Passing learning rate as positional argument
• Using non-existent tf.optimizers.AdamOptimizer

3. What will be the output loss value after one training step using RMSprop optimizer with learning rate 0.01 on a simple linear model trained on data x=[1,2], y=[2,4]? Assume initial weights are zero and mean squared error loss.

medium

A. 0.5

B. 9.5

C. 1.0

D. 4.0

Solution

1. Step 1: Calculate initial prediction and loss

   Initial weights zero means prediction is 0 for inputs. Loss = mean squared error = mean([4,16]) = 10.

2. Step 2: Perform one RMSprop update step

   RMSprop scales update by rms of gradient (first step rms ≈ 0.32*|g|). Gradients ≈[-10,-6] for [w,b], updates ≈[+0.032,+0.032]. New preds ≈[0.063,0.095], new loss ≈9.5.

3. Final Answer:

   9.5 -> Option B

4. Quick Check:

   Loss after step ≈ 9.5 [OK]

Hint: RMSprop first step small due to scaling, loss ~9.5 [OK]

Common Mistakes:

• Expecting sharp loss drop after one step
• Confusing learning rate effect
• Ignoring initial zero weights impact

4. You wrote this code to use Adam optimizer but get an error:

optimizer = tf.optimizers.Adam(lr=0.01)
model.compile(optimizer=optimizer, loss='mse')

What is the likely cause of the error?

medium

A. Model.compile does not accept optimizer objects

B. Adam optimizer does not accept float arguments

C. Loss function 'mse' is invalid

D. Learning rate must be named as learning_rate=0.01

Solution

1. Step 1: Check Adam optimizer argument requirements

   TF2.x Adam expects keyword 'learning_rate=', not TF1.x-style 'lr='.

2. Step 2: Identify error cause in code

   Using lr=0.01 causes TypeError (unexpected keyword). Correct: tf.optimizers.Adam(learning_rate=0.01).

3. Final Answer:

   Learning rate must be named as learning_rate=0.01 -> Option D

4. Quick Check:

   Named argument needed [OK]

Hint: Always name learning_rate in Adam optimizer [OK]

Common Mistakes:

• Using 'lr=0.01' keyword from TF1.x
• Assuming 'mse' is invalid loss
• Thinking optimizer object can't be passed

5. You want to train a model on noisy data that changes over time. Which optimizer is best suited to adapt learning rates per parameter and handle this noise effectively?

hard

A. Adam

B. Gradient Descent with fixed learning rate

C. RMSprop

D. SGD without momentum

Solution

1. Step 1: Understand optimizer strengths for noisy data

   Adam adapts learning rates per parameter and combines momentum and RMSprop ideas, handling noise well.

2. Step 2: Compare with other optimizers

   SGD without momentum and fixed learning rate struggle with noise. RMSprop adapts rates but Adam adds momentum for better stability.

3. Final Answer:

   Adam -> Option A

4. Quick Check:

   Best for noisy data = Adam [OK]

Hint: Adam adapts learning rates and handles noise best [OK]

Common Mistakes:

• Choosing plain SGD for noisy data
• Confusing RMSprop with Adam's momentum
• Ignoring adaptive learning rate benefits