Gradient descent optimization in ML Python - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is gradient descent in simple terms?

Gradient descent is a way to find the lowest point of a hill by taking small steps downhill. In machine learning, it helps find the best model by reducing errors step by step.

beginner

What role does the learning rate play in gradient descent?

The learning rate controls how big each step downhill is. If it's too big, you might jump past the lowest point. If it's too small, it takes a long time to reach the bottom.

intermediate

Explain the difference between batch, stochastic, and mini-batch gradient descent.

Batch gradient descent uses all data to take one step, which is slow but stable. Stochastic uses one data point per step, which is fast but noisy. Mini-batch uses a small group of data points, balancing speed and stability.

intermediate

Why can gradient descent get stuck in local minima?

Sometimes the error surface has many low points. Gradient descent might find a small dip (local minimum) instead of the lowest dip (global minimum), so it stops there and misses the best solution.

advanced

What is the purpose of using momentum in gradient descent?

Momentum helps gradient descent keep moving in the right direction by remembering past steps. It can speed up learning and help jump over small bumps or flat areas.

What does gradient descent aim to minimize?

A The number of data points

B The learning rate

C The model size

D The error or loss function

Gradient descent tries to reduce the error or loss function to improve the model's predictions.

If the learning rate is too high, what can happen?

A The model will always find the global minimum

B The model converges too slowly

C The model might overshoot the minimum and fail to converge

D The model ignores the data

A high learning rate can cause the steps to jump over the minimum, preventing convergence.

Which type of gradient descent updates the model after looking at one data point?

A Stochastic gradient descent

B Mini-batch gradient descent

C Batch gradient descent

D Momentum gradient descent

Stochastic gradient descent updates the model after each single data point.

What problem does momentum help solve in gradient descent?

A Avoiding local minima

B Speeding up convergence and smoothing updates

C Increasing the learning rate automatically

D Reducing the size of the dataset

Momentum helps speed up learning and smooth out the updates by remembering past gradients.

Which of these is NOT a type of gradient descent?

A Random gradient descent

B Stochastic gradient descent

C Batch gradient descent

D Mini-batch gradient descent

Random gradient descent is not a recognized type of gradient descent.

Describe how gradient descent helps a machine learning model learn from data.

Think about walking downhill to find the lowest point.

Explain the differences and trade-offs between batch, stochastic, and mini-batch gradient descent.

Consider how much data is used for each update.