Least squares optimization in SciPy - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of simple least squares fit

What is the output of the following code that fits a line y = mx + c to data points using scipy.optimize.least_squares?

SciPy

import numpy as np
from scipy.optimize import least_squares

def fun(params, x, y):
    m, c = params
    return m * x + c - y

x = np.array([0, 1, 2, 3])
y = np.array([1, 3, 5, 7])

res = least_squares(fun, x0=[0, 0], args=(x, y))
print(res.x.round(2))

A [0.5 1.0]

B [1.0 2.0]

C [2.0 1.0]

D [2.0 0.0]

💡 Hint

Think about the line that best fits points (0,1), (1,3), (2,5), (3,7).

The points lie perfectly on the line y = 2x + 1. The least squares finds slope m=2 and intercept c=1.

❓ data_output

intermediate

Number of iterations in least squares optimization

How many iterations does the least_squares optimizer perform for this problem?

SciPy

import numpy as np
from scipy.optimize import least_squares

def fun(params, x, y):
    m, c = params
    return m * x + c - y

x = np.array([0, 1, 2, 3])
y = np.array([1, 3, 5, 7])

res = least_squares(fun, x0=[0, 0], args=(x, y))
print(res.nfev)

A 3

B 6

C 5

D 4

💡 Hint

Check the attribute that counts function evaluations in the result object.

The optimizer evaluates the function 4 times before converging for this simple problem.

🔧 Debug

advanced

Identify the error in least squares residual function

What error does this code raise when run?

SciPy

import numpy as np
from scipy.optimize import least_squares

def fun(params, x, y):
    m, c = params
    return m * x + c + y

x = np.array([0, 1, 2])
y = np.array([1, 2, 3])

res = least_squares(fun, x0=[0, 0], args=(x, y))

A ValueError: residuals must be difference between model and data

B No error, runs successfully

C RuntimeWarning: overflow encountered in add

D The optimization converges to wrong parameters but no error

💡 Hint

Check if the function returns an array of residuals as expected.

The function returns residuals but adds y instead of subtracting. This does not raise an error but leads to wrong fit.

🧠 Conceptual

advanced

Effect of initial guess on least squares result

Which statement about the initial guess in scipy.optimize.least_squares is true?

A The initial guess affects the speed but not the final solution for convex problems.

B A poor initial guess always causes the optimizer to fail.

C The initial guess does not affect the final result for linear problems.

D The initial guess must be exactly the true parameters to converge.

💡 Hint

Think about convex problems and local minima.

For convex problems like linear least squares, initial guess affects speed but optimizer converges to the same solution.

🚀 Application

expert

Fitting a nonlinear model with least squares

Given data points x and y, which option fits the nonlinear model y = a * exp(b * x) using scipy.optimize.least_squares and returns the optimized parameters?

SciPy

import numpy as np
from scipy.optimize import least_squares

x = np.array([0, 1, 2, 3])
y = np.array([1, 2.7, 7.4, 20.1])

A

def fun(params, x, y):
    a, b = params
    return a * np.exp(b * x) - y
res = least_squares(fun, x0=[1, 0.5], args=(x, y))
print(res.x.round(2))

B

def fun(params, x, y):
    a, b = params
    return y - a * np.exp(b * x)
res = least_squares(fun, x0=[1, 0.5], args=(x, y))
print(res.x.round(2))

C

def fun(params, x, y):
    a, b = params
    return np.log(y) - (a + b * x)
res = least_squares(fun, x0=[1, 0.5], args=(x, y))
print(res.x.round(2))

D

def fun(params, x, y):
    a, b = params
    return a * b * x - y
res = least_squares(fun, x0=[1, 0.5], args=(x, y))
print(res.x.round(2))

💡 Hint

Residuals should be model prediction minus observed data.

Option A correctly defines residuals as model minus data. Option A reverses residuals but least_squares minimizes sum of squares so it still works but is less conventional. Option A tries log transform incorrectly. Option A is wrong model.