Least squares optimization in SciPy - Deep Dive

Overview - Least squares optimization

What is it?

Least squares optimization is a method to find the best fit line or curve to a set of data points by minimizing the sum of the squares of the differences between observed and predicted values. It helps us find parameters that make a model closely match the data. This method is widely used in data fitting, regression, and solving equations approximately. It works by adjusting parameters to reduce the total error in predictions.

Why it matters

Without least squares optimization, we would struggle to find simple models that explain data well, making predictions unreliable. It solves the problem of noisy or imperfect data by finding the best compromise fit. This is crucial in science, engineering, and business where decisions depend on understanding trends and relationships from data. Without it, data analysis would be less precise and more guesswork.

Where it fits

Before learning least squares optimization, you should understand basic algebra, functions, and error concepts. After this, you can explore advanced regression techniques, machine learning models, and nonlinear optimization methods. It fits early in the data modeling journey as a fundamental tool for fitting models to data.

Mental Model

Core Idea

Least squares optimization finds the model parameters that make the total squared difference between predicted and actual data as small as possible.

Think of it like...

Imagine trying to draw a straight line through a scatter of points on paper so that the total squared distance from each point to the line is as small as possible. The line you draw is the best compromise that fits all points together.

Data points: *  *    *  *  *
Best fit line:  ------------------
Error: vertical distances from points to line minimized in squared sum

Build-Up - 7 Steps

1

Foundation

Understanding data points and errors

Concept: Learn what data points are and how errors measure differences between predictions and actual values.

Data points are pairs of inputs and outputs we observe. Errors are the vertical distances between the predicted output from a model and the actual output. Squaring these errors makes all differences positive and emphasizes larger errors.

Result

You can measure how well a model fits data by calculating squared errors for each point.

Understanding errors as distances helps grasp why minimizing their squares leads to the best fit.

2

Foundation

Linear models and parameters

Concept: Introduce simple linear models with parameters that predict outputs from inputs.

A linear model predicts output y from input x using parameters like slope (m) and intercept (b): y = m*x + b. Changing m and b changes the line's position and angle.

Result

You can represent many relationships with simple lines controlled by parameters.

Knowing models depend on parameters sets the stage for adjusting them to fit data.

3

Intermediate

Formulating least squares as optimization

🤔Before reading on: do you think minimizing sum of absolute errors or sum of squared errors is better for fitting? Commit to your answer.

Concept: Express least squares as minimizing the sum of squared errors to find best parameters.

We define a function that sums the squared differences between predicted and actual values for all data points. The goal is to find parameters that make this sum as small as possible. This is an optimization problem.

Result

We have a clear goal: find parameters that minimize total squared error.

Formulating fitting as optimization connects data fitting to powerful mathematical tools.

4

Intermediate

Using scipy.optimize.least_squares

🤔Before reading on: do you think least_squares requires the function to return errors or squared errors? Commit to your answer.

Concept: Learn how to use scipy's least_squares function to solve least squares problems.

scipy.optimize.least_squares takes a function that returns residuals (errors) for each data point and finds parameters minimizing their squares. You provide initial guesses and the function to compute residuals.

Result

You can solve least squares problems easily with code that returns residuals.

Knowing the function returns residuals (not squared) is key to using scipy correctly.

5

Intermediate

Interpreting optimization results

Concept: Understand the output of least squares optimization and how to check fit quality.

The result object contains optimized parameters, success status, and error metrics. You can plot the fitted model against data to visually check fit quality.

Result

You get parameters that best fit data and can evaluate how well the model works.

Interpreting results helps confirm if the model and optimization succeeded.

6

Advanced

Handling nonlinear models and constraints

🤔Before reading on: do you think least squares only works for straight lines? Commit to your answer.

Concept: Extend least squares to nonlinear models and add constraints on parameters.

Least squares can fit curves by defining residuals from nonlinear functions. scipy allows bounds on parameters to keep them realistic. This flexibility lets you model complex relationships.

Result

You can fit curves and control parameter ranges for better models.

Understanding nonlinear fitting expands least squares to many real-world problems.

7

Expert

Numerical stability and algorithm choices

🤔Before reading on: do you think all least squares algorithms perform equally well on all data? Commit to your answer.

Concept: Explore how different algorithms and data scaling affect numerical stability and solution quality.

Least squares solvers use methods like Levenberg-Marquardt or Trust Region Reflective. Some handle large or ill-conditioned data better. Scaling data and choosing algorithms carefully prevents errors and slow convergence.

Result

You achieve reliable, accurate fits even on challenging data sets.

Knowing solver internals and data preparation prevents subtle bugs and improves performance.

Under the Hood

Least squares optimization works by iteratively adjusting parameters to reduce the sum of squared residuals. Internally, algorithms compute the Jacobian matrix of residuals with respect to parameters to guide updates. Methods like Levenberg-Marquardt blend gradient descent and Gauss-Newton steps to balance speed and stability. The process continues until changes are small or a maximum iteration count is reached.

Why designed this way?

The squared error is smooth and differentiable, making it suitable for gradient-based optimization. Early methods used normal equations but were numerically unstable for large or ill-conditioned data. Modern iterative algorithms improve stability and handle nonlinear models. The design balances mathematical tractability, computational efficiency, and robustness.

┌─────────────────────────────┐
│ Start with initial parameters │
└─────────────┬───────────────┘
              │
              ▼
┌─────────────────────────────┐
│ Compute residuals (errors)   │
│ and Jacobian matrix          │
└─────────────┬───────────────┘
              │
              ▼
┌─────────────────────────────┐
│ Update parameters using      │
│ optimization algorithm       │
└─────────────┬───────────────┘
              │
              ▼
┌─────────────────────────────┐
│ Check convergence criteria   │
│ (small change or max steps)  │
└───────┬─────────────┬───────┘
        │             │
        ▼             ▼
   Converged       Not converged
    Output          Repeat steps

Myth Busters - 4 Common Misconceptions

Quick: Does minimizing sum of absolute errors give the same result as minimizing sum of squared errors? Commit to yes or no.

Common Belief: Minimizing absolute errors and squared errors produce the same best fit parameters.

Reality: Minimizing squared errors penalizes large errors more heavily and leads to different fits than minimizing absolute errors.

Why it matters: Choosing the wrong error metric can lead to fits that are less sensitive to outliers or that do not represent the data well.

Quick: Is least squares optimization only for linear models? Commit to yes or no.

Common Belief: Least squares only works for straight lines or linear relationships.

Reality: Least squares can fit nonlinear models by defining residuals from any function and using iterative solvers.

Why it matters: Limiting least squares to linear models restricts its use and misses powerful nonlinear fitting capabilities.

Quick: Does scipy.optimize.least_squares require the function to return squared errors? Commit to yes or no.

Common Belief: The function passed to least_squares must return squared errors for optimization.

Reality: The function must return residuals (errors), not squared errors; the algorithm squares them internally.

Why it matters: Returning squared errors causes incorrect optimization and wrong results.

Quick: Can least squares always find the global best fit? Commit to yes or no.

Common Belief: Least squares optimization always finds the global minimum error solution.

Reality: For nonlinear problems, least squares may find local minima depending on initial guesses.

Why it matters: Without good initial parameters, the solution may be suboptimal and misleading.

Expert Zone

1

The choice of initial parameter guess can drastically affect convergence speed and final solution in nonlinear least squares.

2

Scaling input data and parameters improves numerical stability and prevents solver failures.

3

Different algorithms (e.g., Levenberg-Marquardt vs Trust Region Reflective) have tradeoffs in speed, robustness, and constraint handling.

When NOT to use

Least squares is not ideal when errors are not normally distributed or when outliers dominate; robust regression or other loss functions like Huber loss are better. For discrete or classification problems, other optimization methods apply.

Production Patterns

In real-world systems, least squares is used for sensor calibration, curve fitting in experiments, and parameter estimation in simulations. It is often combined with data preprocessing, parameter constraints, and validation to ensure reliable models.

Connections

Gradient Descent Optimization

Least squares optimization uses gradient-based methods like gradient descent to minimize error functions.

Understanding gradient descent helps grasp how least squares iteratively improves parameter estimates.

Machine Learning Regression

Least squares is the foundation for linear regression models in machine learning.

Knowing least squares clarifies how regression models learn from data by minimizing prediction errors.

Physics: Least Action Principle

Both least squares and least action principles find minimum values of a quantity to explain natural phenomena or data.

Recognizing this connection shows how optimization ideas unify science and data analysis.

Common Pitfalls

#1 Returning squared errors instead of residuals in the function passed to least_squares.

Wrong approach:

def residuals(params, x, y):
return (y - (params[0]*x + params[1]))**2 # wrong: returns squared errors

Correct approach:

def residuals(params, x, y):
return y - (params[0]*x + params[1]) # correct: returns residuals

Root cause:

Misunderstanding that least_squares expects residuals, not squared residuals.

#2 Not providing a reasonable initial guess for parameters in nonlinear fitting.

Wrong approach:

result = least_squares(residuals, x0=[0, 0, 0, 0], args=(x, y)) # poor guess for complex model

Correct approach:

result = least_squares(residuals, x0=[1, 0.5, 0, 0], args=(x, y)) # better initial guess

Root cause:

Ignoring the importance of initial parameters leads to slow or failed convergence.

#3 Ignoring parameter bounds when parameters must be positive or within limits.

Wrong approach:

result = least_squares(residuals, x0=[-1, 2], args=(x, y)) # no bounds, negative parameter allowed

Correct approach:

result = least_squares(residuals, x0=[1, 2], bounds=([0, 0], [np.inf, np.inf]), args=(x, y)) # enforce positivity

Root cause:

Not using bounds causes unrealistic or invalid parameter estimates.

Key Takeaways

Least squares optimization finds parameters that minimize the total squared difference between model predictions and data.

It works by iteratively adjusting parameters using gradient-based algorithms guided by residuals and their derivatives.

scipy.optimize.least_squares requires a function returning residuals, not squared residuals, for correct operation.

Least squares applies to both linear and nonlinear models, with options for parameter constraints and bounds.

Understanding solver choices, initial guesses, and data scaling is essential for reliable and accurate fitting.