Least squares optimization in SciPy - Step-by-Step Execution

Concept Flow - Least squares optimization

Define model function

↓

Provide data points

↓

Define residuals function

↓

Call least_squares optimizer

↓

Optimizer adjusts parameters

↓

Calculate residuals

↓

Minimize sum of squares of residuals

↓

Return optimized parameters

The process starts by defining a model and residuals, then the optimizer adjusts parameters to minimize the sum of squared residuals, returning the best fit.

Execution Sample

SciPy

import numpy as np
from scipy.optimize import least_squares

def model(x, t):
    return x[0] * t + x[1]

def residuals(x, t, y):
    return model(x, t) - y

# Data points
t = np.array([0, 1, 2, 3])
y = np.array([1, 3, 5, 7])

# Initial guess
x0 = np.array([0, 0])

# Run optimizer
result = least_squares(residuals, x0, args=(t, y))
print(result.x)

This code fits a line y = m*t + b to data points by minimizing squared residuals using scipy's least_squares.

Execution Table

Step	Parameters (x)	Residuals	Sum of Squares	Action
1	[0.0, 0.0]	[-1, -3, -5, -7]	84	Initial guess, calculate residuals and sum of squares
2	[1.0, 0.0]	[0, -2, -4, -6]	56	Optimizer updates slope to 1.0
3	[2.0, 0.0]	[1, -1, -3, -5]	36	Optimizer updates slope to 2.0
4	[2.0, 1.0]	[0, 0, 0, 0]	0	Optimizer updates intercept to 1.0
5	[2.0, 1.0]	[0, 0, 0, 0]	0	No further improvement, optimization stops

💡 Sum of squares stops decreasing significantly, optimizer converges at parameters [2.0, 1.0]

Variable Tracker

Variable	Start	After 1	After 2	After 3	After 4	Final
x (parameters)	[0.0, 0.0]	[0.0, 0.0]	[1.0, 0.0]	[2.0, 0.0]	[2.0, 1.0]	[2.0, 1.0]
Residuals	N/A	[-1, -3, -5, -7]	[0, -2, -4, -6]	[1, -1, -3, -5]	[0, 0, 0, 0]	[0, 0, 0, 0]
Sum of Squares	N/A	84	56	36	0	0

Key Moments - 2 Insights

Why do residuals change when parameters change?

Residuals are the differences between model predictions and actual data. When parameters change (see execution_table steps 1 to 4), the model predictions change, so residuals update accordingly.

Why does the optimizer stop at step 5 even though residuals are not zero?

The optimizer stops when it cannot reduce the sum of squares significantly anymore (step 5). Perfect zero residuals are rare; the goal is to minimize error as much as possible.

Visual Quiz - 3 Questions

Look at the execution table, what are the parameters after step 3?

A [1.0, 0.0]

B [2.0, 0.0]

C [0.0, 0.0]

D [2.0, 1.0]

💡 Hint

Check the 'Parameters (x)' column at step 3 in the execution_table.

At which step does the sum of squares first drop below 40?

A Step 3

B Step 4

C Step 2

D Step 5

💡 Hint

Look at the 'Sum of Squares' column in the execution_table for values below 40.

If the initial guess was [1, 1], how would the sum of squares at step 1 change?

A It would be exactly 84

B It would be larger than 84

C It would be smaller than 84

D It would be zero

💡 Hint

Initial residuals depend on how close the guess is to data; [1,1] is closer than [0,0], so sum of squares decreases.

Concept Snapshot

Least squares optimization fits model parameters by minimizing the sum of squared residuals.
Define a model and residuals function.
Use scipy.optimize.least_squares with initial guess and data.
Optimizer iteratively updates parameters to reduce error.
Stops when improvements become minimal.

Full Transcript

Least squares optimization finds the best parameters for a model by minimizing the squared differences between predicted and actual data. We start by defining a model function and a residuals function that calculates errors. Using scipy's least_squares, we provide an initial guess and data points. The optimizer changes parameters step-by-step, recalculating residuals and their sum of squares. When the sum of squares stops decreasing significantly, the optimizer stops and returns the best parameters found. This process helps fit models like lines to data in a way that minimizes overall error.

Practice

1. What is the main goal of using scipy.optimize.least_squares in data fitting?

easy

A. To sort the data points in ascending order

B. To maximize the difference between the model and data

C. To find parameters that minimize the difference between the model and data

D. To randomly select parameters for the model

Solution

1. Step 1: Understand the purpose of least squares

   Least squares optimization aims to find parameters that reduce the error between predicted and actual data.

2. Step 2: Connect to scipy.optimize.least_squares

   This function specifically minimizes the sum of squared residuals, which are differences between model and data.

3. Final Answer:

   To find parameters that minimize the difference between the model and data -> Option C

4. Quick Check:

   Least squares = minimize difference [OK]

Hint: Least squares means minimizing errors, not maximizing [OK]

Common Mistakes:

• Thinking it maximizes difference
• Confusing with sorting or random selection
• Assuming it changes data order

2. Which of the following is the correct way to call scipy.optimize.least_squares with a residual function fun and initial guess x0?

easy

A. least_squares(fun)

B. least_squares(x0, fun)

C. least_squares(fun=x0, x0=fun)

D. least_squares(fun, x0)

Solution

1. Step 1: Check the function signature

   The correct call is least_squares(fun, x0) where fun is the residual function and x0 is the initial guess.

2. Step 2: Verify argument order

   Arguments must be in order: first the function, then the initial guess.

3. Final Answer:

   least_squares(fun, x0) -> Option D

4. Quick Check:

   Function first, initial guess second [OK]

Hint: Function first, initial guess second in call [OK]

Common Mistakes:

• Swapping argument order
• Using keyword arguments incorrectly
• Omitting the initial guess

3. What will be the output of this code snippet?

import numpy as np
from scipy.optimize import least_squares

def residuals(x):
    return np.array([2*x[0] - 4, x[1] + 3])

result = least_squares(residuals, [0, 0])
print(result.x)

medium

A. [4.0, -3.0]

B. [2.0, -3.0]

C. [0.0, 0.0]

D. [-2.0, 3.0]

Solution

1. Step 1: Solve residual equations for zero residuals

   Set residuals to zero: 2*x0 - 4 = 0 => x0 = 2; x1 + 3 = 0 => x1 = -3.

2. Step 2: Confirm least_squares finds these values

   The optimizer finds x = [2, -3] minimizing residuals to zero.

3. Final Answer:

   [2.0, -3.0] -> Option B

4. Quick Check:

   2*2-4=0 and -3+3=0 [OK]

Hint: Set residuals to zero and solve for variables [OK]

Common Mistakes:

• Not solving equations correctly
• Confusing signs in residuals
• Assuming initial guess is output

4. Identify the error in this code snippet using least_squares:

from scipy.optimize import least_squares

def fun(x):
    return x**2 - 4

result = least_squares(fun)
print(result.x)

medium

A. Missing initial guess argument in least_squares call

B. Residual function returns scalar instead of array

C. Function fun should return x**2 + 4

D. Print statement syntax is incorrect

Solution

1. Step 1: Check least_squares function call

   The call lacks the required initial guess argument x0.

2. Step 2: Confirm residual function and print are correct

   The residual function returns an array-like (scalar is acceptable as 1D array), and print syntax is valid.

3. Final Answer:

   Missing initial guess argument in least_squares call -> Option A

4. Quick Check:

   least_squares needs initial guess [OK]

Hint: Always provide initial guess to least_squares [OK]

Common Mistakes:

• Forgetting initial guess
• Thinking scalar residuals cause error
• Misreading print syntax

5. You want to fit a line y = mx + c to data points x = [1, 2, 3] and y = [2, 3, 5] using least_squares. Which residual function correctly represents the difference between observed and predicted values?

hard

A. def residuals(p):\n m, c = p\n return [(m*x[i] + c) - y[i] for i in range(len(x))]

B. def residuals(p):\n m, c = p\n return [y[i] - (m*x[i] + c) for i in range(len(x))]

C. def residuals(p):\n m, c = p\n return [y[i] + (m*x[i] + c) for i in range(len(x))]

D. def residuals(p):\n m, c = p\n return [(m*x[i] - c) - y[i] for i in range(len(x))]

Solution

1. Step 1: Understand residual definition

   Residuals are predicted minus observed values: (model - data).

2. Step 2: Check each function

   def residuals(p):\n m, c = p\n return [(m*x[i] + c) - y[i] for i in range(len(x))] returns (m*x + c) - y, matching predicted minus observed.

3. Final Answer:

   def residuals(p):\n m, c = p\n return [(m*x[i] + c) - y[i] for i in range(len(x))] -> Option A

4. Quick Check:

   Residual = predicted - observed [OK]

Hint: Residual = predicted minus observed values [OK]

Common Mistakes:

• Swapping predicted and observed in residuals
• Adding instead of subtracting values
• Incorrect sign on intercept