Least squares (least_squares) in SciPy - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of scipy.optimize.least_squares with linear residuals

What is the output of the following code snippet using scipy.optimize.least_squares to fit a line y = mx + b to data points?

SciPy

import numpy as np
from scipy.optimize import least_squares

def residuals(params, x, y):
    m, b = params
    return m * x + b - y

x_data = np.array([0, 1, 2, 3])
y_data = np.array([1, 3, 5, 7])

result = least_squares(residuals, x0=[0, 0], args=(x_data, y_data))
print(result.x)

A [3.0, 0.0]

B [1.0, 2.0]

C [0.5, 1.0]

D [2.0, 1.0]

💡 Hint

Think about the line that fits points (0,1), (1,3), (2,5), (3,7) exactly.

The data points lie perfectly on the line y = 2x + 1. The least squares method finds parameters m=2 and b=1.

❓ data_output

intermediate

Number of iterations in least_squares optimization

After running the following code, what is the value of result.nfev (number of function evaluations)?

SciPy

import numpy as np
from scipy.optimize import least_squares

def residuals(p, x, y):
    return p[0] * x + p[1] - y

x = np.linspace(0, 10, 5)
y = 3 * x + 4 + np.random.normal(0, 0.1, size=x.size)

result = least_squares(residuals, x0=[0, 0], args=(x, y))
print(result.nfev)

A 10

B 4

C 6

D 1

💡 Hint

The solver usually needs multiple evaluations to converge, but not too many for a simple linear fit.

The solver typically evaluates the residual function about 6 times for this simple problem before converging.

🔧 Debug

advanced

Identify the error in least_squares residual function

What error will this code raise when run?

SciPy

import numpy as np
from scipy.optimize import least_squares

def residuals(params, x, y):
    m, b = params
    return m * x + b + y  # Incorrect sign here

x = np.array([1, 2, 3])
y = np.array([2, 4, 6])

result = least_squares(residuals, x0=[0, 0], args=(x, y))

A No error, but wrong fit result

B TypeError due to incompatible operations

C ValueError due to shape mismatch

D RuntimeWarning due to overflow

💡 Hint

Check the residuals formula carefully for correctness.

The code runs without error but the residuals are incorrect because it adds y instead of subtracting, leading to a wrong fit.

🧠 Conceptual

advanced

Effect of jac='3-point' in least_squares

What does setting jac='3-point' do in scipy.optimize.least_squares?

A Uses an analytical Jacobian provided by the user

B Uses a numerical approximation of the Jacobian using three points per variable

C Disables Jacobian calculation for faster runtime

D Uses a random Jacobian matrix for stochastic optimization

💡 Hint

Think about numerical differentiation methods.

The '3-point' option uses a finite difference method with three points to approximate the Jacobian numerically.

🚀 Application

expert

Predict parameter values from least_squares output

Given this output from least_squares: result.x = [1.5, -0.5], and residuals defined as m * x + b - y, what is the predicted y value for x=4?

A 5.5

B 6.5

C 7.5

D 4.5

💡 Hint

Use the formula y = m*x + b with given parameters.

With m=1.5 and b=-0.5, y = 1.5*4 - 0.5 = 6 - 0.5 = 5.5.

Practice

1. What is the main purpose of using scipy.optimize.least_squares in data science?

easy

A. To find the best fit parameters by minimizing the difference between model predictions and data

B. To sort data points in ascending order

C. To calculate the mean of a dataset

D. To generate random numbers for simulations

Solution

1. Step 1: Understand the purpose of least squares

   Least squares is used to find parameters that minimize the error between a model and observed data.

2. Step 2: Match the purpose with the options

   Only To find the best fit parameters by minimizing the difference between model predictions and data describes minimizing differences to find best fit parameters.

3. Final Answer:

   To find the best fit parameters by minimizing the difference between model predictions and data -> Option A

4. Quick Check:

   Least squares = minimize error [OK]

Hint: Least squares minimizes errors to fit data best [OK]

Common Mistakes:

• Confusing least squares with sorting or averaging
• Thinking it generates random data
• Assuming it calculates statistics like mean

2. Which of the following is the correct way to call scipy.optimize.least_squares with a residual function fun and initial guess x0?

easy

A. least_squares(fun=x0, x0=fun)

B. least_squares(x0, fun)

C. least_squares(fun, x0)

D. least_squares(x0)

Solution

1. Step 1: Recall the function signature

   The correct call is least_squares(fun, x0) where fun is the residual function and x0 is the initial guess.

2. Step 2: Check each option

   least_squares(fun, x0) matches the correct order and parameters. Others have wrong order or missing arguments.

3. Final Answer:

   least_squares(fun, x0) -> Option C

4. Quick Check:

   Function first, initial guess second [OK]

Hint: Function first, initial guess second in least_squares call [OK]

Common Mistakes:

• Swapping the order of arguments
• Passing only one argument
• Using keyword arguments incorrectly

3. What will be the output of this code snippet?

import numpy as np
from scipy.optimize import least_squares

def residuals(x):
    return np.array([2*x[0] - 4, x[1] + 3])

result = least_squares(residuals, x0=[0, 0])
print(result.x)

medium

A. [-2.0, 3.0]

B. [4.0, 3.0]

C. [0.0, 0.0]

D. [2.0, -3.0]

Solution

1. Step 1: Understand the residual function

   The residuals are [2*x0 - 4, x1 + 3]. We want to find x that makes residuals close to zero.

2. Step 2: Solve equations for zero residuals

   Set 2*x0 - 4 = 0 => x0 = 2; and x1 + 3 = 0 => x1 = -3.

3. Final Answer:

   [2.0, -3.0] -> Option D

4. Quick Check:

   Zero residuals at x=[2, -3] [OK]

Hint: Set residuals to zero and solve for variables [OK]

Common Mistakes:

• Not solving residual equations correctly
• Confusing signs in equations
• Assuming initial guess is the answer

4. Identify the error in this code using least_squares:

import numpy as np
from scipy.optimize import least_squares

def residuals(x):
    return 2*x - 5

result = least_squares(residuals, x0=3)
print(result.x)

medium

A. Initial guess x0 should be a list or array, not a scalar

B. Residual function returns a scalar instead of an array

C. Missing import statement for numpy

D. least_squares requires a Jacobian function

Solution

1. Step 1: Check residual function output

   The residual function returns 2*x - 5, which is a scalar, but least_squares expects an array-like residual.

2. Step 2: Verify other parts

   x0 as scalar is allowed; numpy is imported; Jacobian is optional.

3. Final Answer:

   Residual function returns a scalar instead of an array -> Option B

4. Quick Check:

   Residuals must be array-like [OK]

Hint: Residuals must be array, not single number [OK]

Common Mistakes:

• Returning scalar residual instead of array
• Thinking initial guess must be array
• Assuming Jacobian is mandatory

5. You have noisy data points for a line: x = [0,1,2,3], y = [1.1, 2.0, 2.9, 4.2]. Using least_squares, which residual function best fits a line model y = m*x + c to estimate m and c?

hard

A. def residuals(p): return y - (p[0]*x + p[1])

B. def residuals(p): return p[0]*x + p[1]

C. def residuals(p): return (p[0]*x + p[1]) * y

D. def residuals(p): return y / (p[0]*x + p[1])

Solution

1. Step 1: Understand residuals for least squares

   Residuals are differences between observed y and model predictions m*x + c.

2. Step 2: Check residual function forms

   def residuals(p): return y - (p[0]*x + p[1]) returns y - model prediction (m*x + c), the standard residuals to minimize. def residuals(p): return p[0]*x + p[1] returns only the model predictions without subtracting y, so it minimizes the sum of squared model values instead of fitting errors.

3. Step 3: Eliminate incorrect options

   Options C and D multiply or divide, which is incorrect for residuals.

4. Final Answer:

   def residuals(p): return y - (p[0]*x + p[1]) -> Option A

5. Quick Check:

   Residual = observed - predicted [OK]

Hint: Residual = observed minus predicted values [OK]

Common Mistakes:

• Using multiplication or division instead of subtraction
• Forgetting to subtract the observed values
• Ignoring residuals should be array differences