Gradient descent optimization in ML Python

Introduction

Gradient descent helps a model learn by slowly improving its guesses to make fewer mistakes.

When training a model to find the best settings that reduce errors.

When you want to improve predictions step by step using data.

When adjusting weights in neural networks to get better results.

When minimizing a function that measures how wrong the model is.

When you want a simple way to optimize many types of models.

Syntax

ML Python

for each step:
    calculate gradient of loss with respect to parameters
    update parameters by moving opposite to gradient scaled by learning rate

The 'gradient' shows the direction to change parameters to reduce error.

The 'learning rate' controls how big each step is; too big can overshoot, too small can be slow.

Examples

This updates the weights by moving them opposite to the gradient scaled by the learning rate.

ML Python

weights = weights - learning_rate * gradient

A simple loop showing repeated updates to weights to reduce loss over 100 steps.

ML Python

for epoch in range(100):
    predictions = model(inputs)
    loss = loss_function(predictions, targets)
    gradient = compute_gradient(loss, weights)
    weights -= learning_rate * gradient

Sample Model

This code fits a line y=2x+1 using gradient descent to find weight and bias that minimize the error.

ML Python

import numpy as np

# Simple linear regression using gradient descent

# Data: y = 2x + 1
x = np.array([1, 2, 3, 4, 5], dtype=float)
y = 2 * x + 1

# Initialize parameters
weight = 0.0
bias = 0.0
learning_rate = 0.1

for epoch in range(20):
    # Predictions
    y_pred = weight * x + bias
    
    # Calculate error
    error = y_pred - y
    
    # Calculate gradients
    dweight = (2/len(x)) * np.dot(error, x)
    dbias = (2/len(x)) * np.sum(error)
    
    # Update parameters
    weight -= learning_rate * dweight
    bias -= learning_rate * dbias
    
    # Calculate loss (mean squared error)
    loss = np.mean(error ** 2)
    print(f"Epoch {epoch+1}: loss={loss:.4f}, weight={weight:.4f}, bias={bias:.4f}")

Important Notes

Choosing the right learning rate is important: too high can make training unstable, too low can make it slow.

Gradient descent can get stuck in flat or bad spots; more advanced methods help avoid this.

Batch gradient descent uses all data each step; stochastic uses one example; mini-batch uses small groups.

Summary

Gradient descent helps models learn by adjusting parameters to reduce errors.

It works by moving parameters opposite to the gradient of the loss.

Small steps controlled by learning rate improve the model gradually.