Gradient descent optimization in ML Python - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is gradient descent in simple terms?

Gradient descent is a way to find the lowest point of a hill by taking small steps downhill. In machine learning, it helps find the best model by reducing errors step by step.

beginner

What role does the learning rate play in gradient descent?

The learning rate controls how big each step downhill is. If it's too big, you might jump past the lowest point. If it's too small, it takes a long time to reach the bottom.

intermediate

Explain the difference between batch, stochastic, and mini-batch gradient descent.

Batch gradient descent uses all data to take one step, which is slow but stable. Stochastic uses one data point per step, which is fast but noisy. Mini-batch uses a small group of data points, balancing speed and stability.

intermediate

Why can gradient descent get stuck in local minima?

Sometimes the error surface has many low points. Gradient descent might find a small dip (local minimum) instead of the lowest dip (global minimum), so it stops there and misses the best solution.

advanced

What is the purpose of using momentum in gradient descent?

Momentum helps gradient descent keep moving in the right direction by remembering past steps. It can speed up learning and help jump over small bumps or flat areas.

What does gradient descent aim to minimize?

A The number of data points

B The learning rate

C The model size

D The error or loss function

Gradient descent tries to reduce the error or loss function to improve the model's predictions.

If the learning rate is too high, what can happen?

A The model will always find the global minimum

B The model converges too slowly

C The model might overshoot the minimum and fail to converge

D The model ignores the data

A high learning rate can cause the steps to jump over the minimum, preventing convergence.

Which type of gradient descent updates the model after looking at one data point?

A Stochastic gradient descent

B Mini-batch gradient descent

C Batch gradient descent

D Momentum gradient descent

Stochastic gradient descent updates the model after each single data point.

What problem does momentum help solve in gradient descent?

A Avoiding local minima

B Speeding up convergence and smoothing updates

C Increasing the learning rate automatically

D Reducing the size of the dataset

Momentum helps speed up learning and smooth out the updates by remembering past gradients.

Which of these is NOT a type of gradient descent?

A Random gradient descent

B Stochastic gradient descent

C Batch gradient descent

D Mini-batch gradient descent

Random gradient descent is not a recognized type of gradient descent.

Describe how gradient descent helps a machine learning model learn from data.

Think about walking downhill to find the lowest point.

Explain the differences and trade-offs between batch, stochastic, and mini-batch gradient descent.

Consider how much data is used for each update.

Practice

1. What is the main goal of gradient descent in machine learning?

easy

A. To minimize the error or loss function

B. To increase the number of model parameters

C. To randomly change model weights

D. To maximize the training time

Solution

1. Step 1: Understand gradient descent purpose

   Gradient descent is used to find the best model parameters by reducing errors.

2. Step 2: Connect to error minimization

   The algorithm updates parameters to minimize the loss function, which measures error.

3. Final Answer:

   To minimize the error or loss function -> Option A

4. Quick Check:

   Goal = minimize loss [OK]

Hint: Gradient descent reduces errors step-by-step [OK]

Common Mistakes:

• Thinking it increases parameters
• Believing it changes weights randomly
• Confusing it with training time increase

2. Which of the following is the correct update rule for a parameter \(\theta\) using gradient descent with learning rate \(\alpha\) and gradient \(\nabla J(\theta)\)?

easy

A. \(\theta = \theta - \alpha \nabla J(\theta)\)

B. \(\theta = \theta + \alpha \nabla J(\theta)\)

C. \(\theta = \alpha \nabla J(\theta) - \theta\)

D. \(\theta = \theta \times \alpha \nabla J(\theta)\)

Solution

1. Step 1: Recall gradient descent formula

   Gradient descent updates parameters by moving opposite to the gradient to reduce loss.

2. Step 2: Identify correct sign and terms

   The update is \(\theta = \theta - \alpha \nabla J(\theta)\), subtracting the gradient scaled by learning rate.

3. Final Answer:

   \(\theta = \theta - \alpha \nabla J(\theta)\) -> Option A

4. Quick Check:

   Update = subtract gradient [OK]

Hint: Subtract gradient times learning rate [OK]

Common Mistakes:

• Adding instead of subtracting gradient
• Multiplying parameters incorrectly
• Swapping terms order

3. Consider this Python code snippet for one step of gradient descent on parameter \(\theta\):

theta = 2.0
learning_rate = 0.1
gradient = 3.0

theta = theta - learning_rate * gradient
print(round(theta, 2))

What will be the printed output?

medium

A. -0.1

B. 1.3

C. 2.3

D. 1.7

Solution

1. Step 1: Calculate the update amount

   Update = learning_rate * gradient = 0.1 * 3.0 = 0.3

2. Step 2: Apply update to theta

   New theta = 2.0 - 0.3 = 1.7

3. Step 3: Check printed value

   Printed value is rounded to 2 decimals: 1.7

4. Final Answer:

   1.7 -> Option D

5. Quick Check:

   2.0 - 0.3 = 1.7 [OK]

Hint: Subtract learning_rate * gradient from theta [OK]

Common Mistakes:

• Adding instead of subtracting update
• Mixing gradient and learning rate
• Rounding incorrectly

4. The following code tries to perform one gradient descent step but has a bug:

theta = 5.0
learning_rate = 0.01
gradient = -4.0

theta = theta + learning_rate * gradient
print(theta)

What is the bug and how to fix it?

medium

A. No bug; code is correct

B. Bug: learning rate too small; fix by increasing it

C. Bug: sign error; fix by changing '+' to '-' in update

D. Bug: gradient sign wrong; fix by negating gradient

Solution

1. Step 1: Recall standard gradient descent formula

   The correct update is always \(\theta = \theta - \alpha \nabla J(\theta)\).

2. Step 2: Identify the bug in code

   The code uses '+' instead of '-', performing gradient ascent. With gradient = -4.0, code updates to 5.0 - 0.04 = 4.96 (wrong direction); correct is 5.0 + 0.04 = 5.04.

3. Final Answer:

   Bug: sign error; fix by changing '+' to '-' in update -> Option C

4. Quick Check:

   Always subtract lr * gradient [OK]

Hint: Always use '-' for gradient descent [OK]

Common Mistakes:

• Thinking '+' works for negative gradients
• Confusing ascent (+) with descent (-)
• Ignoring standard update rule

5. You run gradient descent with a very large learning rate \(\alpha = 10\) on a simple quadratic loss \(J(\theta) = (\theta - 3)^2\). What is the most likely outcome after a few steps starting from \(\theta=0\)?

hard

A. The parameter \(\theta\) converges smoothly to 3

B. The parameter \(\theta\) oscillates or diverges away from 3

C. The parameter \(\theta\) stays at 0 without change

D. The parameter \(\theta\) becomes negative infinity

Solution

1. Step 1: Understand effect of large learning rate

   A large learning rate causes big jumps in parameter updates, possibly overshooting the minimum.

2. Step 2: Analyze behavior on quadratic loss

   Since \(J(\theta) = (\theta - 3)^2\), gradient is \(2(\theta - 3)\). Large steps can cause \(\theta\) to jump past 3 repeatedly, causing oscillation or divergence.

3. Final Answer:

   The parameter \(\theta\) oscillates or diverges away from 3 -> Option B

4. Quick Check:

   Large learning rate causes oscillation [OK]

Hint: Too big steps cause oscillation, not smooth learning [OK]

Common Mistakes:

• Assuming large learning rate helps convergence
• Thinking parameter stays fixed
• Confusing divergence with negative infinity