Optimizers (SGD, Adam, RMSprop) in TensorFlow - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is the main role of an optimizer in machine learning?

An optimizer helps the model learn by adjusting its internal settings (weights) to reduce errors and improve predictions.

beginner

How does the SGD optimizer update model weights?

SGD (Stochastic Gradient Descent) updates weights by moving them a small step opposite to the error direction, using a fixed learning rate.

intermediate

What makes Adam optimizer different from SGD?

Adam combines ideas from momentum and RMSprop, adapting learning rates for each weight individually, which helps faster and more stable learning.

intermediate

Why is RMSprop useful for training neural networks?

RMSprop adjusts the learning rate for each weight based on recent gradients, helping the model learn well even when gradients vary a lot.

beginner

Which optimizer would you choose for a simple linear model and why?

SGD is often chosen for simple models because it is straightforward and effective when the learning rate is well tuned.

Which optimizer adapts the learning rate for each parameter individually?

A SGD

B Adam

C Batch Gradient Descent

D None of the above

Adam adjusts learning rates for each parameter based on past gradients, unlike SGD which uses a fixed rate.

What does SGD stand for?

A Stochastic Gradient Descent

B Simple Gradient Descent

C Sequential Gradient Descent

D Standard Gradient Descent

SGD means Stochastic Gradient Descent, which updates weights using small random batches.

Which optimizer uses a moving average of squared gradients to adjust learning rates?

A SGD

B Adam

C RMSprop

D Momentum

RMSprop uses a moving average of squared gradients to adapt learning rates for each parameter.

Why might Adam be preferred over SGD?

A It adapts learning rates and converges faster

B It is simpler to implement

C It requires no learning rate

D It uses less memory

Adam adapts learning rates for each parameter and often converges faster than SGD.

Which optimizer is best described as 'simple and effective with a fixed learning rate'?

A Adagrad

B RMSprop

C Adam

D SGD

SGD uses a fixed learning rate and is simple and effective for many problems.

Explain how the Adam optimizer works and why it might be better than SGD for some problems.

Think about how Adam changes learning rates for each weight and uses past gradients.

Describe the differences between SGD, RMSprop, and Adam optimizers in simple terms.

Focus on how each optimizer changes learning rates and uses past information.