Why optimization finds best solutions in SciPy - Visual Breakdown

Concept Flow - Why optimization finds best solutions

Define objective function

↓

Choose initial guess

↓

Evaluate function at guess

↓

Check if solution is good enough?

Yes→Stop and return solution

No↓

Update guess to improve solution

↓

Repeat evaluation and update

Optimization tries different guesses to find the best value of a function by repeating evaluation and improving guesses until it finds the best solution.

Execution Sample

SciPy

from scipy.optimize import minimize

def f(x):
    return (x - 3)**2

result = minimize(f, x0=0)
print(result.x)

This code finds the value of x that makes (x-3)^2 smallest using scipy.optimize.minimize.

Execution Table

Step	Current x	Function f(x)	Improvement?	Action
1	0	9	Yes	Try new x closer to 3
2	1.5	2.25	Yes	Try new x closer to 3
3	2.25	0.5625	Yes	Try new x closer to 3
4	2.625	0.140625	Yes	Try new x closer to 3
5	2.8125	0.03515625	Yes	Try new x closer to 3
6	2.90625	0.0087890625	Yes	Try new x closer to 3
7	2.953125	0.002197265625	Yes	Try new x closer to 3
8	2.9765625	0.00054931640625	Yes	Try new x closer to 3
9	2.98828125	0.0001373291015625	Yes	Try new x closer to 3
10	2.994140625	3.4332275390625e-05	Yes	Try new x closer to 3
11	2.9970703125	8.58306884765625e-06	Yes	Try new x closer to 3
12	2.99853515625	2.1457672119140625e-06	Yes	Try new x closer to 3
13	2.999267578125	5.364418029785156e-07	Yes	Try new x closer to 3
14	2.9996337890625	1.341104507446289e-07	Yes	Try new x closer to 3
15	2.99981689453125	3.3527612686157227e-08	Yes	Try new x closer to 3
16	2.999908447265625	8.381903171539307e-09	Yes	Try new x closer to 3
17	2.9999542236328125	2.0954757928848267e-09	Yes	Try new x closer to 3
18	2.9999771118164062	5.238689482212067e-10	Yes	Try new x closer to 3
19	2.999988555908203	1.3096723705530167e-10	Yes	Try new x closer to 3
20	2.9999942779541016	3.2741809263825417e-11	Yes	Try new x closer to 3
21	2.999997138977051	8.185452316174072e-12	Yes	Try new x closer to 3
22	3.0	0.0	No	Stop optimization

💡 At step 22, function value is 0, no improvement possible, optimization stops.

Variable Tracker

Variable	Start	After 1	After 2	After 3	After 4	After 5	After 6	After 7	After 8	After 9	After 10	After 11	After 12	After 13	After 14	After 15	After 16	After 17	After 18	After 19	After 20	After 21	Final
x	0	1.5	2.25	2.625	2.8125	2.90625	2.953125	2.9765625	2.98828125	2.994140625	2.9970703125	2.99853515625	2.999267578125	2.9996337890625	2.99981689453125	2.999908447265625	2.9999542236328125	2.9999771118164062	2.999988555908203	2.9999942779541016	2.999997138977051	3.0	3.0
f(x)	9	2.25	0.5625	0.140625	0.03515625	0.0087890625	0.002197265625	0.00054931640625	0.0001373291015625	3.4332275390625e-05	8.58306884765625e-06	2.1457672119140625e-06	5.364418029785156e-07	1.341104507446289e-07	3.3527612686157227e-08	8.381903171539307e-09	2.0954757928848267e-09	5.238689482212067e-10	1.3096723705530167e-10	3.2741809263825417e-11	8.185452316174072e-12	0.0	0.0

Key Moments - 3 Insights

Why does the function value get smaller with each step?

Because the optimizer updates x to values closer to 3, which reduces (x-3)^2, as shown in the execution_table where f(x) decreases each step.

Why does the optimization stop at step 22?

At step 22, f(x) is 0, meaning the best possible value is found. The condition 'No improvement?' is true, so the optimizer stops, as seen in the exit_note.

Why start with x=0 and not directly at 3?

The optimizer needs a starting guess to begin searching. Starting at 0 shows how it improves step by step toward 3, as tracked in variable_tracker.

Visual Quiz - 3 Questions

Look at the execution_table at step 5. What is the value of f(x)?

A 0.140625

B 0.5625

C 0.03515625

D 2.25

💡 Hint

Check the 'Function f(x)' column in execution_table row for step 5.

At which step does the optimizer stop because no improvement is possible?

A Step 22

B Step 18

C Step 20

D Step 15

💡 Hint

Look at the 'Improvement?' and 'Action' columns in execution_table for the stopping step.

If the initial guess x0 was 5 instead of 0, how would the first function value f(x) compare?

A It would be 9

B It would be 4

C It would be 1

D It would be 0

💡 Hint

Calculate (5-3)^2 to find f(x) at x=5, compare with initial f(x) in variable_tracker.

Concept Snapshot

Optimization finds the best solution by:
- Defining a function to minimize or maximize
- Starting with an initial guess
- Evaluating the function at guesses
- Updating guesses to improve the function value
- Stopping when no better solution is found
Use scipy.optimize.minimize for easy optimization in Python.

Full Transcript

Optimization is a process to find the best value of a function. We start by defining the function we want to minimize, like (x-3)^2. We pick a starting guess, here x=0. The optimizer checks the function value at this guess, then tries new guesses closer to the best value. Each step improves the guess, making the function value smaller. When the function value cannot get any smaller, the optimizer stops and returns the best x found. This step-by-step process is shown in the execution_table and variable_tracker. Beginners often wonder why the function value decreases each step, why the optimizer stops, and why we need a starting guess. These are explained by how the optimizer moves closer to the minimum and stops when no improvement is possible.