Least squares optimization in SciPy - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of simple least squares fit

What is the output of the following code that fits a line y = mx + c to data points using scipy.optimize.least_squares?

SciPy

import numpy as np
from scipy.optimize import least_squares

def fun(params, x, y):
    m, c = params
    return m * x + c - y

x = np.array([0, 1, 2, 3])
y = np.array([1, 3, 5, 7])

res = least_squares(fun, x0=[0, 0], args=(x, y))
print(res.x.round(2))

A [0.5 1.0]

B [1.0 2.0]

C [2.0 1.0]

D [2.0 0.0]

💡 Hint

Think about the line that best fits points (0,1), (1,3), (2,5), (3,7).

The points lie perfectly on the line y = 2x + 1. The least squares finds slope m=2 and intercept c=1.

❓ data_output

intermediate

Number of iterations in least squares optimization

How many iterations does the least_squares optimizer perform for this problem?

SciPy

import numpy as np
from scipy.optimize import least_squares

def fun(params, x, y):
    m, c = params
    return m * x + c - y

x = np.array([0, 1, 2, 3])
y = np.array([1, 3, 5, 7])

res = least_squares(fun, x0=[0, 0], args=(x, y))
print(res.nfev)

A 3

B 6

C 5

D 4

💡 Hint

Check the attribute that counts function evaluations in the result object.

The optimizer evaluates the function 4 times before converging for this simple problem.

🔧 Debug

advanced

Identify the error in least squares residual function

What error does this code raise when run?

SciPy

import numpy as np
from scipy.optimize import least_squares

def fun(params, x, y):
    m, c = params
    return m * x + c + y

x = np.array([0, 1, 2])
y = np.array([1, 2, 3])

res = least_squares(fun, x0=[0, 0], args=(x, y))

A ValueError: residuals must be difference between model and data

B No error, runs successfully

C RuntimeWarning: overflow encountered in add

D The optimization converges to wrong parameters but no error

💡 Hint

Check if the function returns an array of residuals as expected.

The function returns residuals but adds y instead of subtracting. This does not raise an error but leads to wrong fit.

🧠 Conceptual

advanced

Effect of initial guess on least squares result

Which statement about the initial guess in scipy.optimize.least_squares is true?

A The initial guess affects the speed but not the final solution for convex problems.

B A poor initial guess always causes the optimizer to fail.

C The initial guess does not affect the final result for linear problems.

D The initial guess must be exactly the true parameters to converge.

💡 Hint

Think about convex problems and local minima.

For convex problems like linear least squares, initial guess affects speed but optimizer converges to the same solution.

🚀 Application

expert

Fitting a nonlinear model with least squares

Given data points x and y, which option fits the nonlinear model y = a * exp(b * x) using scipy.optimize.least_squares and returns the optimized parameters?

SciPy

import numpy as np
from scipy.optimize import least_squares

x = np.array([0, 1, 2, 3])
y = np.array([1, 2.7, 7.4, 20.1])

A

def fun(params, x, y):
    a, b = params
    return a * np.exp(b * x) - y
res = least_squares(fun, x0=[1, 0.5], args=(x, y))
print(res.x.round(2))

B

def fun(params, x, y):
    a, b = params
    return y - a * np.exp(b * x)
res = least_squares(fun, x0=[1, 0.5], args=(x, y))
print(res.x.round(2))

C

def fun(params, x, y):
    a, b = params
    return np.log(y) - (a + b * x)
res = least_squares(fun, x0=[1, 0.5], args=(x, y))
print(res.x.round(2))

D

def fun(params, x, y):
    a, b = params
    return a * b * x - y
res = least_squares(fun, x0=[1, 0.5], args=(x, y))
print(res.x.round(2))

💡 Hint

Residuals should be model prediction minus observed data.

Option A correctly defines residuals as model minus data. Option A reverses residuals but least_squares minimizes sum of squares so it still works but is less conventional. Option A tries log transform incorrectly. Option A is wrong model.

Practice

1. What is the main goal of using scipy.optimize.least_squares in data fitting?

easy

A. To sort the data points in ascending order

B. To maximize the difference between the model and data

C. To find parameters that minimize the difference between the model and data

D. To randomly select parameters for the model

Solution

1. Step 1: Understand the purpose of least squares

   Least squares optimization aims to find parameters that reduce the error between predicted and actual data.

2. Step 2: Connect to scipy.optimize.least_squares

   This function specifically minimizes the sum of squared residuals, which are differences between model and data.

3. Final Answer:

   To find parameters that minimize the difference between the model and data -> Option C

4. Quick Check:

   Least squares = minimize difference [OK]

Hint: Least squares means minimizing errors, not maximizing [OK]

Common Mistakes:

• Thinking it maximizes difference
• Confusing with sorting or random selection
• Assuming it changes data order

2. Which of the following is the correct way to call scipy.optimize.least_squares with a residual function fun and initial guess x0?

easy

A. least_squares(fun)

B. least_squares(x0, fun)

C. least_squares(fun=x0, x0=fun)

D. least_squares(fun, x0)

Solution

1. Step 1: Check the function signature

   The correct call is least_squares(fun, x0) where fun is the residual function and x0 is the initial guess.

2. Step 2: Verify argument order

   Arguments must be in order: first the function, then the initial guess.

3. Final Answer:

   least_squares(fun, x0) -> Option D

4. Quick Check:

   Function first, initial guess second [OK]

Hint: Function first, initial guess second in call [OK]

Common Mistakes:

• Swapping argument order
• Using keyword arguments incorrectly
• Omitting the initial guess

3. What will be the output of this code snippet?

import numpy as np
from scipy.optimize import least_squares

def residuals(x):
    return np.array([2*x[0] - 4, x[1] + 3])

result = least_squares(residuals, [0, 0])
print(result.x)

medium

A. [4.0, -3.0]

B. [2.0, -3.0]

C. [0.0, 0.0]

D. [-2.0, 3.0]

Solution

1. Step 1: Solve residual equations for zero residuals

   Set residuals to zero: 2*x0 - 4 = 0 => x0 = 2; x1 + 3 = 0 => x1 = -3.

2. Step 2: Confirm least_squares finds these values

   The optimizer finds x = [2, -3] minimizing residuals to zero.

3. Final Answer:

   [2.0, -3.0] -> Option B

4. Quick Check:

   2*2-4=0 and -3+3=0 [OK]

Hint: Set residuals to zero and solve for variables [OK]

Common Mistakes:

• Not solving equations correctly
• Confusing signs in residuals
• Assuming initial guess is output

4. Identify the error in this code snippet using least_squares:

from scipy.optimize import least_squares

def fun(x):
    return x**2 - 4

result = least_squares(fun)
print(result.x)

medium

A. Missing initial guess argument in least_squares call

B. Residual function returns scalar instead of array

C. Function fun should return x**2 + 4

D. Print statement syntax is incorrect

Solution

1. Step 1: Check least_squares function call

   The call lacks the required initial guess argument x0.

2. Step 2: Confirm residual function and print are correct

   The residual function returns an array-like (scalar is acceptable as 1D array), and print syntax is valid.

3. Final Answer:

   Missing initial guess argument in least_squares call -> Option A

4. Quick Check:

   least_squares needs initial guess [OK]

Hint: Always provide initial guess to least_squares [OK]

Common Mistakes:

• Forgetting initial guess
• Thinking scalar residuals cause error
• Misreading print syntax

5. You want to fit a line y = mx + c to data points x = [1, 2, 3] and y = [2, 3, 5] using least_squares. Which residual function correctly represents the difference between observed and predicted values?

hard

A. def residuals(p):\n m, c = p\n return [(m*x[i] + c) - y[i] for i in range(len(x))]

B. def residuals(p):\n m, c = p\n return [y[i] - (m*x[i] + c) for i in range(len(x))]

C. def residuals(p):\n m, c = p\n return [y[i] + (m*x[i] + c) for i in range(len(x))]

D. def residuals(p):\n m, c = p\n return [(m*x[i] - c) - y[i] for i in range(len(x))]

Solution

1. Step 1: Understand residual definition

   Residuals are predicted minus observed values: (model - data).

2. Step 2: Check each function

   def residuals(p):\n m, c = p\n return [(m*x[i] + c) - y[i] for i in range(len(x))] returns (m*x + c) - y, matching predicted minus observed.

3. Final Answer:

   def residuals(p):\n m, c = p\n return [(m*x[i] + c) - y[i] for i in range(len(x))] -> Option A

4. Quick Check:

   Residual = predicted - observed [OK]

Hint: Residual = predicted minus observed values [OK]

Common Mistakes:

• Swapping predicted and observed in residuals
• Adding instead of subtracting values
• Incorrect sign on intercept