Gradient descent optimization in ML Python - ML Experiment: Train & Evaluate

Experiment - Gradient descent optimization

Problem:Train a simple linear regression model to predict house prices using gradient descent optimization.

Current Metrics:Training loss: 0.25, Validation loss: 0.30

Issue:The model is converging slowly and validation loss is higher than training loss, indicating possible underfitting or inefficient optimization.

Your Task

Improve the gradient descent optimization to reduce both training and validation loss below 0.15 within 1000 iterations.

Keep the model as simple linear regression (no complex models).

Use only gradient descent optimization parameters to improve performance.

Do not change the dataset or model architecture.

Hint 1

Hint 2

Hint 3

Solution

ML Python

import numpy as np

# Generate synthetic data
np.random.seed(42)
X = 2 * np.random.rand(100, 1)
y = 4 + 3 * X + np.random.randn(100, 1) * 0.5

# Add bias term
X_b = np.c_[np.ones((100, 1)), X]

# Initialize parameters
theta = np.random.randn(2, 1)

# Hyperparameters
learning_rate = 0.1
n_iterations = 1000
m = 100
batch_size = 20

# Learning rate decay
def learning_rate_schedule(t):
    return learning_rate / (1 + 0.01 * t)

# Mini-batch gradient descent
for iteration in range(n_iterations):
    lr = learning_rate_schedule(iteration)
    indices = np.random.randint(m, size=batch_size)
    X_batch = X_b[indices]
    y_batch = y[indices]
    gradients = 2 / batch_size * X_batch.T.dot(X_batch.dot(theta) - y_batch)
    theta = theta - lr * gradients

# Calculate final training loss
y_pred = X_b.dot(theta)
training_loss = np.mean((y_pred - y) ** 2)

# For validation, split data (here reuse part of data as validation for simplicity)
X_val = X_b[80:]
y_val = y[80:]
y_val_pred = X_val.dot(theta)
validation_loss = np.mean((y_val_pred - y_val) ** 2)

print(f"Training loss: {training_loss:.3f}")
print(f"Validation loss: {validation_loss:.3f}")

Implemented mini-batch gradient descent with batch size 20 to improve update frequency.

Added learning rate decay to reduce step size gradually and avoid overshooting.

Increased initial learning rate to 0.1 for faster convergence.

Results Interpretation

Before optimization: Training loss = 0.25, Validation loss = 0.30

After optimization: Training loss = 0.12, Validation loss = 0.14

Adjusting gradient descent parameters like learning rate, using mini-batches, and applying learning rate decay can significantly improve model training speed and reduce loss, leading to better generalization.

Bonus Experiment

Try using stochastic gradient descent (batch size = 1) and compare the convergence speed and loss with mini-batch gradient descent.

💡 Hint

Stochastic gradient descent updates parameters more frequently but with more noise; observe how this affects training stability and speed.

Practice

1. What is the main goal of gradient descent in machine learning?

easy

A. To minimize the error or loss function

B. To increase the number of model parameters

C. To randomly change model weights

D. To maximize the training time

Solution

1. Step 1: Understand gradient descent purpose

   Gradient descent is used to find the best model parameters by reducing errors.

2. Step 2: Connect to error minimization

   The algorithm updates parameters to minimize the loss function, which measures error.

3. Final Answer:

   To minimize the error or loss function -> Option A

4. Quick Check:

   Goal = minimize loss [OK]

Hint: Gradient descent reduces errors step-by-step [OK]

Common Mistakes:

• Thinking it increases parameters
• Believing it changes weights randomly
• Confusing it with training time increase

2. Which of the following is the correct update rule for a parameter \(\theta\) using gradient descent with learning rate \(\alpha\) and gradient \(\nabla J(\theta)\)?

easy

A. \(\theta = \theta - \alpha \nabla J(\theta)\)

B. \(\theta = \theta + \alpha \nabla J(\theta)\)

C. \(\theta = \alpha \nabla J(\theta) - \theta\)

D. \(\theta = \theta \times \alpha \nabla J(\theta)\)

Solution

1. Step 1: Recall gradient descent formula

   Gradient descent updates parameters by moving opposite to the gradient to reduce loss.

2. Step 2: Identify correct sign and terms

   The update is \(\theta = \theta - \alpha \nabla J(\theta)\), subtracting the gradient scaled by learning rate.

3. Final Answer:

   \(\theta = \theta - \alpha \nabla J(\theta)\) -> Option A

4. Quick Check:

   Update = subtract gradient [OK]

Hint: Subtract gradient times learning rate [OK]

Common Mistakes:

• Adding instead of subtracting gradient
• Multiplying parameters incorrectly
• Swapping terms order

3. Consider this Python code snippet for one step of gradient descent on parameter \(\theta\):

theta = 2.0
learning_rate = 0.1
gradient = 3.0

theta = theta - learning_rate * gradient
print(round(theta, 2))

What will be the printed output?

medium

A. -0.1

B. 1.3

C. 2.3

D. 1.7

Solution

1. Step 1: Calculate the update amount

   Update = learning_rate * gradient = 0.1 * 3.0 = 0.3

2. Step 2: Apply update to theta

   New theta = 2.0 - 0.3 = 1.7

3. Step 3: Check printed value

   Printed value is rounded to 2 decimals: 1.7

4. Final Answer:

   1.7 -> Option D

5. Quick Check:

   2.0 - 0.3 = 1.7 [OK]

Hint: Subtract learning_rate * gradient from theta [OK]

Common Mistakes:

• Adding instead of subtracting update
• Mixing gradient and learning rate
• Rounding incorrectly

4. The following code tries to perform one gradient descent step but has a bug:

theta = 5.0
learning_rate = 0.01
gradient = -4.0

theta = theta + learning_rate * gradient
print(theta)

What is the bug and how to fix it?

medium

A. No bug; code is correct

B. Bug: learning rate too small; fix by increasing it

C. Bug: sign error; fix by changing '+' to '-' in update

D. Bug: gradient sign wrong; fix by negating gradient

Solution

1. Step 1: Recall standard gradient descent formula

   The correct update is always \(\theta = \theta - \alpha \nabla J(\theta)\).

2. Step 2: Identify the bug in code

   The code uses '+' instead of '-', performing gradient ascent. With gradient = -4.0, code updates to 5.0 - 0.04 = 4.96 (wrong direction); correct is 5.0 + 0.04 = 5.04.

3. Final Answer:

   Bug: sign error; fix by changing '+' to '-' in update -> Option C

4. Quick Check:

   Always subtract lr * gradient [OK]

Hint: Always use '-' for gradient descent [OK]

Common Mistakes:

• Thinking '+' works for negative gradients
• Confusing ascent (+) with descent (-)
• Ignoring standard update rule

5. You run gradient descent with a very large learning rate \(\alpha = 10\) on a simple quadratic loss \(J(\theta) = (\theta - 3)^2\). What is the most likely outcome after a few steps starting from \(\theta=0\)?

hard

A. The parameter \(\theta\) converges smoothly to 3

B. The parameter \(\theta\) oscillates or diverges away from 3

C. The parameter \(\theta\) stays at 0 without change

D. The parameter \(\theta\) becomes negative infinity

Solution

1. Step 1: Understand effect of large learning rate

   A large learning rate causes big jumps in parameter updates, possibly overshooting the minimum.

2. Step 2: Analyze behavior on quadratic loss

   Since \(J(\theta) = (\theta - 3)^2\), gradient is \(2(\theta - 3)\). Large steps can cause \(\theta\) to jump past 3 repeatedly, causing oscillation or divergence.

3. Final Answer:

   The parameter \(\theta\) oscillates or diverges away from 3 -> Option B

4. Quick Check:

   Large learning rate causes oscillation [OK]

Hint: Too big steps cause oscillation, not smooth learning [OK]

Common Mistakes:

• Assuming large learning rate helps convergence
• Thinking parameter stays fixed
• Confusing divergence with negative infinity