Minimizing multivariate functions (minimize) in SciPy - Step-by-Step Execution

Concept Flow - Minimizing multivariate functions (minimize)

Define function f(x, y)

↓

Choose initial guess (x0, y0)

↓

Call minimize(f, x0)

↓

Algorithm iterates to find minimum

↓

Check convergence

↓

Return minimum

↓

End

We start by defining the function to minimize, pick a starting point, then use minimize to find the lowest value by iterating until it converges.

Execution Sample

SciPy

from scipy.optimize import minimize

def f(x):
    return (x[0]-1)**2 + (x[1]+2)**2

result = minimize(f, [0, 0])
print(result.x)

This code finds the minimum of a function with two variables starting from (0,0).

Execution Table

Step	Current guess (x, y)	Function value f(x,y)	Action	Notes
1	[0.0, 0.0]	5.0	Evaluate f at start	Initial guess
2	[0.5, -1.0]	1.25	Move towards minimum	Improved guess
3	[0.75, -1.5]	0.3125	Move closer	Better guess
4	[0.9, -1.8]	0.058	Move closer	Near minimum
5	[0.99, -1.98]	0.0004	Move closer	Very close
6	[1.0, -2.0]	0.0	Converged	Minimum found
7	-	-	Stop	Convergence criteria met

💡 Algorithm stops when function value change is below threshold and guess is near minimum.

Variable Tracker

Variable	Start	After 1	After 2	After 3	After 4	After 5	Final
x	[0.0, 0.0]	[0.5, -1.0]	[0.75, -1.5]	[0.9, -1.8]	[0.99, -1.98]	[1.0, -2.0]	[1.0, -2.0]
f(x)	5.0	1.25	0.3125	0.058	0.0004	0.0	0.0

Key Moments - 3 Insights

Why does the algorithm start from [0, 0] and not directly at the minimum?

The algorithm needs a starting guess to begin searching. It improves the guess step-by-step as shown in the execution_table rows 1 to 6.

What tells the algorithm to stop iterating?

When the function value changes very little between steps and the guess is close to the minimum, the algorithm stops, as seen in step 6 and 7.

Why is the function value decreasing at each step?

Because the algorithm moves the guess towards points where the function value is lower, aiming to find the minimum, as shown in the decreasing f(x) values in variable_tracker.

Visual Quiz - 3 Questions

Look at the execution_table at step 3. What is the current guess and function value?

A [0.5, -1.0], 1.25

B [0.9, -1.8], 0.058

C [0.75, -1.5], 0.3125

D [1.0, -2.0], 0.0

💡 Hint

Check the row labeled step 3 in the execution_table.

At which step does the algorithm converge to the minimum?

A Step 6

B Step 5

C Step 4

D Step 7

💡 Hint

Look for the step where the function value is 0.0 and action is 'Converged' in execution_table.

If the initial guess was changed to [2, 2], what would likely happen in the execution_table?

A The function values would increase at each step.

B The steps would start from [2, 2] with a higher function value and then decrease.

C The algorithm would immediately find the minimum at step 1.

D The algorithm would not run because initial guess must be [0, 0].

💡 Hint

Initial guess affects starting point and function value, as shown in variable_tracker start values.

Concept Snapshot

Use scipy.optimize.minimize to find minimum of multivariate functions.
Define function f(x) returning scalar.
Provide initial guess x0 as list or array.
minimize iterates guesses to reduce f(x).
Stops when convergence criteria met.
Result contains minimum location in result.x.

Full Transcript

We start by defining a function with multiple variables that we want to minimize. We pick a starting guess for the variables. Then we call scipy.optimize.minimize with the function and the guess. The algorithm evaluates the function at the guess, then moves the guess step-by-step to lower function values. It repeats this until it finds a point where the function value cannot be lowered further. This point is the minimum. The process is shown step-by-step in the execution table and variable tracker. Beginners often wonder why we need a starting guess and how the algorithm knows when to stop. The starting guess is necessary to begin searching, and the algorithm stops when changes become very small. The final result gives the minimum point found.

Practice

1. What is the main purpose of scipy.optimize.minimize in data science?

easy

A. To find the minimum value of a multivariate function

B. To sort data in ascending order

C. To calculate the mean of a dataset

D. To generate random numbers

Solution

1. Step 1: Understand the function's goal

   scipy.optimize.minimize is designed to find the lowest point of a function with multiple variables.

2. Step 2: Compare options with the function's purpose

   Sorting, calculating mean, or generating random numbers are unrelated to minimizing functions.

3. Final Answer:

   To find the minimum value of a multivariate function -> Option A

4. Quick Check:

   Minimize means find lowest point [OK]

Hint: Minimize means find lowest point of a function [OK]

Common Mistakes:

• Confusing minimize with sorting or statistics
• Thinking it generates random data
• Assuming it calculates averages

2. Which of the following is the correct way to call scipy.optimize.minimize for a function f with initial guess x0?

easy

A. minimize(f, x0, method='max')

B. minimize(f, x0)

C. minimize(f(x0))

D. minimize(x0, f)

Solution

1. Step 1: Recall the function signature

   The correct call is minimize(fun, x0) where fun is the function and x0 is the starting guess.

2. Step 2: Check each option

   minimize(f, x0) matches the correct order. minimize(x0, f) reverses arguments. minimize(f(x0)) calls the function instead of passing it. minimize(f, x0, method='max') uses an invalid method 'max'.

3. Final Answer:

   minimize(f, x0) -> Option B

4. Quick Check:

   Function first, guess second [OK]

Hint: Pass function first, then initial guess [OK]

Common Mistakes:

• Swapping function and initial guess
• Calling function instead of passing it
• Using invalid method names

3. What will be the output of this code snippet?

from scipy.optimize import minimize

def f(x):
    return (x[0]-1)**2 + (x[1]+2)**2

result = minimize(f, [0, 0])
print(result.x.round(2))

medium

A. [1.00, 2.00]

B. [0.00, 0.00]

C. [-1.00, 2.00]

D. [1.00, -2.00]

Solution

1. Step 1: Understand the function to minimize

   The function is (x0-1)^2 + (x1+2)^2, which is smallest when x0=1 and x1=-2.

2. Step 2: Check the minimize result

   Starting from [0,0], minimize finds the point closest to the minimum, which is [1, -2].

3. Final Answer:

   [1.00, -2.00] -> Option D

4. Quick Check:

   Minimum at x=1, y=-2 [OK]

Hint: Minimum at points making squared terms zero [OK]

Common Mistakes:

• Confusing signs inside the function
• Expecting initial guess as output
• Misreading the function formula

4. Identify the error in this code that tries to minimize a function:

from scipy.optimize import minimize

def f(x):
    return x[0]**2 + x[1]**2

result = minimize(f, 0)
print(result.x)

medium

A. Initial guess should be a list or array, not a single number

B. Function f should return a list, not a number

C. minimize requires method='BFGS' explicitly

D. print statement should be inside a function

Solution

1. Step 1: Check the initial guess type

   The function expects a vector input like [x0, x1], but the initial guess is a single number 0, which is invalid.

2. Step 2: Understand the correct input format

   Initial guess must be a list or array with the same dimension as the function input, e.g., [0, 0].

3. Final Answer:

   Initial guess should be a list or array, not a single number -> Option A

4. Quick Check:

   Input vector must match function variables [OK]

Hint: Initial guess must match function input shape [OK]

Common Mistakes:

• Passing scalar instead of vector as initial guess
• Thinking function output must be list
• Assuming method must be specified always

5. You want to minimize the function f(x, y) = (x-3)^2 + (y+1)^2 but only allow y to be between -2 and 0. Which code correctly applies this constraint using scipy.optimize.minimize?

hard

A. minimize(lambda v: (v[0]-3)**2 + (v[1]+1)**2, [0, 0], constraints={'type':'ineq', 'fun': lambda v: v[1] + 2})

B. minimize(lambda v: (v[0]-3)**2 + (v[1]+1)**2, [0, 0], bounds=[(-2, 0), (None, None)])

C. minimize(lambda v: (v[0]-3)**2 + (v[1]+1)**2, [0, 0], bounds=[(None, None), (-2, 0)])

D. minimize(lambda v: (v[0]-3)**2 + (v[1]+1)**2, [0, 0], constraints={'type':'eq', 'fun': lambda v: v[1] + 1})

Solution

1. Step 1: Understand bounds parameter

   Bounds are given as a list of (min, max) tuples for each variable. Here, x has no bounds, y is between -2 and 0.

2. Step 2: Match bounds to variables

   minimize(lambda v: (v[0]-3)**2 + (v[1]+1)**2, [0, 0], bounds=[(None, None), (-2, 0)]) correctly sets bounds as [(None, None), (-2, 0)] meaning x unbounded, y bounded. minimize(lambda v: (v[0]-3)**2 + (v[1]+1)**2, [0, 0], bounds=[(-2, 0), (None, None)]) reverses bounds. Options A and D use constraints incorrectly for this case.

3. Final Answer:

   minimize(lambda v: (v[0]-3)**2 + (v[1]+1)**2, [0, 0], bounds=[(None, None), (-2, 0)]) -> Option C

4. Quick Check:

   Bounds list matches variable order [OK]

Hint: Bounds list order matches variables order [OK]

Common Mistakes:

• Swapping bounds order for variables
• Using constraints instead of bounds incorrectly
• Setting equality constraint instead of bounds