Optimization callbacks and monitoring in SciPy - Mini Project: Build & Apply

Optimization Callbacks and Monitoring with SciPy

📖 Scenario: You are working on a simple optimization problem where you want to find the minimum of a mathematical function. To understand how the optimization progresses, you want to monitor the values at each step using a callback function.

🎯 Goal: Build a Python script that uses SciPy's minimize function to find the minimum of a quadratic function. You will create a callback function to monitor and store the values of the variable during optimization, then print the collected values.

📋 What You'll Learn

Create a quadratic function f(x) = (x-3)^2 + 4

Create a list called history to store values of x during optimization

Define a callback function store_history that appends the current x value to history

Use scipy.optimize.minimize with method 'BFGS' to minimize f starting from x=0

Pass the callback function to minimize to monitor optimization

Print the history list after optimization completes

💡 Why This Matters

🌍 Real World

Monitoring optimization progress helps in tuning algorithms and understanding how solutions improve step-by-step in machine learning and engineering problems.

💼 Career

Knowing how to use callbacks in optimization is useful for roles in data science, machine learning engineering, and scientific computing where optimization is common.

1

Create the quadratic function

Create a function called f that takes a variable x and returns the value of the quadratic function (x - 3)**2 + 4.

SciPy

# Define the quadratic function f(x)
# Your code here

Hint

Use def to define the function and return the expression (x - 3)**2 + 4.

2

Create a list to store optimization history

Create an empty list called history that will store the values of x during optimization.

SciPy

def f(x):
    return (x - 3)**2 + 4

# Create an empty list called history
# Your code here

Hint

Use history = [] to create an empty list.

3

Define the callback function to store values

Define a function called store_history that takes a parameter xk and appends xk to the history list.

SciPy

def f(x):
    return (x - 3)**2 + 4

history = []

# Define callback function store_history(xk)
# Your code here

Hint

Define a function with def store_history(xk): and use history.append(xk) inside it.

4

Run optimization and print history

Import minimize from scipy.optimize. Use minimize to minimize the function f starting from x0=[0] with method 'BFGS'. Pass the callback function store_history to monitor optimization. Finally, print the history list.

SciPy

def f(x):
    return (x - 3)**2 + 4

history = []

def store_history(xk):
    history.append(xk)

# Import minimize and run optimization with callback
# Print the history list
# Your code here

Hint

Use from scipy.optimize import minimize. Call minimize(f, x0=[0], method='BFGS', callback=store_history). Then print history.

Practice

1. What is the main purpose of using a callback function during optimization in scipy.optimize?

easy

A. To monitor and control the optimization process step-by-step

B. To automatically fix errors in the optimization function

C. To speed up the optimization by parallel processing

D. To save the final result to a file

Solution

1. Step 1: Understand the role of callbacks in optimization

   Callbacks are functions called at each iteration to observe or influence the optimization process.

2. Step 2: Identify the main use of callbacks

   They allow monitoring progress, logging, or stopping optimization early based on conditions.

3. Final Answer:

   To monitor and control the optimization process step-by-step -> Option A

4. Quick Check:

   Callbacks = monitor/control optimization [OK]

Hint: Callbacks watch optimization progress stepwise [OK]

Common Mistakes:

• Thinking callbacks fix errors automatically
• Assuming callbacks speed up optimization
• Believing callbacks save results directly

2. Which of the following is the correct way to define a callback function for scipy.optimize.minimize that prints the current parameter values at each iteration?

easy

A. def callback(): print(f"Current params: {xk}")

B. def callback(xk): return xk

C. def callback(xk, fk): print(f"Current params: {fk}")

D. def callback(xk): print(f"Current params: {xk}")

Solution

1. Step 1: Check callback function signature for scipy.optimize.minimize

   The callback receives one argument: the current parameter vector xk.

2. Step 2: Verify the function prints current parameters correctly

   def callback(xk): print(f"Current params: {xk}") defines callback(xk) and prints xk, which is correct.

3. Final Answer:

   def callback(xk): print(f"Current params: {xk}") -> Option D

4. Quick Check:

   Callback signature = one argument (xk) [OK]

Hint: Callback gets current params as single argument [OK]

Common Mistakes:

• Defining callback without parameters
• Using wrong parameter names or extra parameters
• Returning values instead of printing

3. Given the following code snippet, what will be printed during the optimization?

from scipy.optimize import minimize

def callback(xk):
    print(f"Step: {xk[0]:.2f}, {xk[1]:.2f}")

def func(x):
    return (x[0]-1)**2 + (x[1]-2)**2

res = minimize(func, [0, 0], callback=callback)

medium

A. Multiple lines showing parameter values approaching (1.00, 2.00)

B. Only one line showing initial parameters [0.00, 0.00]

C. No output because callback is ignored

D. Error because callback function has wrong signature

Solution

1. Step 1: Understand the callback usage in minimize

   The callback prints the current parameters at each iteration during optimization.

2. Step 2: Analyze the expected output

   Since the function minimizes distance to (1,2), parameters will update stepwise, printing multiple lines approaching (1.00, 2.00).

3. Final Answer:

   Multiple lines showing parameter values approaching (1.00, 2.00) -> Option A

4. Quick Check:

   Callback prints each step params [OK]

Hint: Callback prints each iteration's parameters [OK]

Common Mistakes:

• Assuming callback prints only once
• Thinking callback is ignored by minimize
• Believing callback signature is incorrect here

4. You wrote this callback to stop optimization early when the first parameter exceeds 0.5:

def callback(xk):
    if xk[0] > 0.5:
        return True

But the optimization does not stop early. What is the likely problem?

medium

A. The callback must raise an exception to stop optimization

B. The callback function must return False to stop optimization

C. Returning True does not stop optimization in scipy.optimize.minimize

D. The callback must be passed as a keyword argument named 'stop_callback'

Solution

1. Step 1: Understand callback behavior in scipy.optimize.minimize

   Callbacks can monitor progress but returning True does not stop the optimization.

2. Step 2: Identify correct way to stop optimization early

   To stop early, you must raise an exception or use other control mechanisms; returning True is ignored.

3. Final Answer:

   Returning True does not stop optimization in scipy.optimize.minimize -> Option C

4. Quick Check:

   Return True ≠ stop optimization [OK]

Hint: Returning True in callback won't stop optimization [OK]

Common Mistakes:

• Expecting return True to stop optimization
• Using wrong callback argument name
• Not raising exception to stop optimization

5. You want to log the optimization progress to a list and stop optimization if the function value goes below 0.01. Which callback implementation correctly achieves this with scipy.optimize.minimize?

hard

A. log = []
def callback(xk):
val = (xk[0]-1)**2 + (xk[1]-2)**2
log.append(val)
if val < 0.01:
return True

res = minimize(lambda x: (x[0]-1)**2 + (x[1]-2)**2, [0,0], callback=callback)

B. log = []
def callback(xk):
val = (xk[0]-1)**2 + (xk[1]-2)**2
log.append(val)
if val < 0.01:
raise StopIteration

res = minimize(lambda x: (x[0]-1)**2 + (x[1]-2)**2, [0,0], callback=callback)

C. log = []
def callback(xk, fk):
log.append(fk)
if fk < 0.01:
raise StopIteration

res = minimize(lambda x: (x[0]-1)**2 + (x[1]-2)**2, [0,0], callback=callback)

D. log = []
def callback(xk):
val = (xk[0]-1)**2 + (xk[1]-2)**2
log.append(val)
if val < 0.01:
return False

res = minimize(lambda x: (x[0]-1)**2 + (x[1]-2)**2, [0,0], callback=callback)

Solution

1. Step 1: Understand callback signature and logging

   The callback receives current parameters xk. We compute function value manually and append to log.

2. Step 2: Stopping optimization early

   Returning True or False does not stop optimization; raising an exception like StopIteration is a valid way.

3. Step 3: Verify options

   log = [] def callback(xk): val = (xk[0]-1)**2 + (xk[1]-2)**2 log.append(val) if val < 0.01: raise StopIteration res = minimize(lambda x: (x[0]-1)**2 + (x[1]-2)**2, [0,0], callback=callback) correctly logs values and raises StopIteration when value < 0.01. log = [] def callback(xk): val = (xk[0]-1)**2 + (xk[1]-2)**2 log.append(val) if val < 0.01: return True res = minimize(lambda x: (x[0]-1)**2 + (x[1]-2)**2, [0,0], callback=callback) returns True (ignored). log = [] def callback(xk, fk): log.append(fk) if fk < 0.01: raise StopIteration res = minimize(lambda x: (x[0]-1)**2 + (x[1]-2)**2, [0,0], callback=callback) has wrong callback signature (two args). log = [] def callback(xk): val = (xk[0]-1)**2 + (xk[1]-2)**2 log.append(val) if val < 0.01: return False res = minimize(lambda x: (x[0]-1)**2 + (x[1]-2)**2, [0,0], callback=callback) returns False (ignored).

4. Final Answer:

   log = [] def callback(xk): val = (xk[0]-1)**2 + (xk[1]-2)**2 log.append(val) if val < 0.01: raise StopIteration res = minimize(lambda x: (x[0]-1)**2 + (x[1]-2)**2, [0,0], callback=callback) -> Option B

5. Quick Check:

   Raise exception to stop + log values [OK]

Hint: Raise exception in callback to stop optimization [OK]

Common Mistakes:

• Returning True or False expecting to stop optimization
• Using wrong callback signature with two arguments
• Not computing function value inside callback