Sparse iterative solvers (gmres, cg) in SciPy - Step-by-Step Execution

Concept Flow - Sparse iterative solvers (gmres, cg)

Start with sparse matrix A and vector b

↓

Choose solver: GMRES or CG

↓

Initialize guess x0 (usually zeros)

↓

Iterative loop: compute residual r = b - A*x

↓

Check residual norm < tolerance?

Yes→Stop, solution x found

No↓

Update solution x using solver step

↓

Repeat iterative loop

The solver starts with a guess and improves it step-by-step until the solution fits the equation well enough.

Execution Sample

SciPy

import numpy as np
from scipy.sparse import diags
from scipy.sparse.linalg import cg

A = diags([1, 2, 1], [-1, 0, 1], shape=(4,4))
b = np.array([1, 2, 2, 1])
x, info = cg(A, b)

This code solves Ax = b using the Conjugate Gradient method on a sparse matrix A.

Execution Table

Iteration	Residual Norm	Condition (residual < tol)	Action	Approximate Solution x
0	3.16	False	Start with x=0, compute initial residual	[0.0, 0.0, 0.0, 0.0]
1	0.18	False	CG step: update x and compute new residual	[0.28, 0.56, 0.56, 0.28]
2	0.00	True	CG step: residual below tolerance, stop	[0.20, 0.60, 0.60, 0.20]

💡 Residual norm reached zero at iteration 2, solution found.

Variable Tracker

Variable	Initial	After 1	After 2
x (solution)	[0.0, 0.0, 0.0, 0.0]	[0.28, 0.56, 0.56, 0.28]	[0.20, 0.60, 0.60, 0.20]
Residual norm	3.16	0.18	0.00

Key Moments - 2 Insights

Why does the residual norm start high and then decrease?

Because the initial guess x=0 is usually far from the true solution, so the difference b - A*x is large. Each iteration improves x, reducing the residual norm as shown in the execution_table rows 0 to 2.

What does it mean when the residual norm becomes zero?

It means the current solution x satisfies the equation Ax = b perfectly within the tolerance, so the solver stops. See execution_table row 2 where the condition becomes True.

Visual Quiz - 3 Questions

Look at the execution table, what is the approximate solution x after iteration 1?

A [0.20, 0.60, 0.60, 0.20]

B [0.28, 0.56, 0.56, 0.28]

C [0.0, 0.0, 0.0, 0.0]

D [1.0, 1.0, 1.0, 1.0]

💡 Hint

Check the 'Approximate Solution x' column at iteration 1 in the execution_table.

At which iteration does the residual norm first become less than 1?

A Iteration 0

B Iteration 2

C Iteration 1

D Never

💡 Hint

Look at the 'Residual Norm' column in the execution_table rows.

If the initial guess x was closer to the true solution, how would the residual norm at iteration 1 change?

A It would be smaller

B It would be larger

C It would be zero

D It would not change

💡 Hint

Refer to variable_tracker showing residual norm changes starting from initial guess.

Concept Snapshot

Sparse iterative solvers like GMRES and CG solve Ax=b for large sparse A.
They start with a guess x0 and improve it iteratively.
Each step reduces the residual r = b - A*x.
Stop when residual norm is below tolerance.
Useful for big sparse systems where direct methods are slow.

Full Transcript

Sparse iterative solvers such as GMRES and CG start with a sparse matrix A and a vector b. They pick an initial guess for the solution x, often zeros. Then they repeatedly compute the residual, which measures how far Ax is from b. If the residual is too large, they update x to get closer to the true solution. This loop continues until the residual is small enough, meaning the solution is good. The example code uses CG to solve a small sparse system, showing how x and residual norm change each iteration until the solution is found.

Practice

1. Which of the following statements about scipy.sparse.linalg.cg is true?

easy

A. cg is slower than gmres for all matrices.

B. cg can solve any linear system regardless of matrix properties.

C. cg uses dense matrix methods internally.

D. cg requires the matrix to be symmetric and positive definite.

Solution

1. Step 1: Understand the requirements of cg

   The conjugate gradient method (cg) is designed for symmetric positive definite matrices only.

2. Step 2: Compare with other options

   cg cannot solve any matrix (B is wrong), it is often faster than gmres for suitable matrices (A is wrong), and it uses sparse methods, not dense (D is wrong).

3. Final Answer:

   cg requires the matrix to be symmetric and positive definite. -> Option D

4. Quick Check:

   cg needs symmetric positive definite matrix [OK]

Hint: Remember: CG needs symmetric positive definite matrices [OK]

Common Mistakes:

• Thinking CG works for any matrix
• Confusing CG with GMRES
• Assuming CG uses dense matrix methods

2. Which is the correct way to import the GMRES solver from SciPy?

easy

A. import scipy.linalg.gmres

B. from scipy.sparse.linalg import gmres

C. from scipy.linalg import gmres

D. import gmres from scipy.sparse

Solution

1. Step 1: Recall the module location of GMRES

   The GMRES solver is in scipy.sparse.linalg, so it must be imported from there.

2. Step 2: Check the import syntax

   Correct Python import syntax for a function is from module import function. from scipy.sparse.linalg import gmres matches this and the correct module.

3. Final Answer:

   from scipy.sparse.linalg import gmres -> Option B

4. Quick Check:

   Correct import syntax and module [OK]

Hint: Import gmres from scipy.sparse.linalg using 'from ... import' [OK]

Common Mistakes:

• Importing from scipy.linalg instead of sparse.linalg
• Using incorrect import syntax
• Trying to import gmres directly from scipy.sparse

3. What will be the output of the following code snippet?

import numpy as np
from scipy.sparse.linalg import cg
from scipy.sparse import diags

A = diags([1, 2, 1], [-1, 0, 1]).toarray()
b = np.array([4, 6, 4])
x, info = cg(A, b)
print(np.round(x, 2))

medium

A. [1. 2. 1.]

B. [2. 1. 2.]

C. [0. 0. 0.]

D. Error due to matrix not positive definite

Solution

1. Step 1: Analyze the matrix and vector

   The matrix A is tridiagonal with diagonals [1,2,1], which is symmetric positive definite. Vector b is [4,6,4].

2. Step 2: Solve using conjugate gradient

   Using cg, the solution x satisfies Ax = b. The solution is approximately [1, 2, 1].

3. Final Answer:

   [1. 2. 1.] -> Option A

4. Quick Check:

   cg solves Ax=b with symmetric positive definite A [OK]

Hint: Check matrix symmetry and positive definiteness before cg [OK]

Common Mistakes:

• Assuming cg fails on this matrix
• Confusing the solution vector with b
• Not rounding output before comparing

4. Identify the error in this code using cg solver:

import numpy as np
from scipy.sparse.linalg import cg

A = np.array([[0, 1], [1, 0]])
b = np.array([1, 2])
x, info = cg(A, b)
print(x)

medium

A. Matrix A is not symmetric positive definite, so cg will fail.

B. Vector b has wrong shape.

C. cg function is not imported correctly.

D. No error; code runs fine.

Solution

1. Step 1: Check matrix properties

   Matrix A = [[0,1],[1,0]] is symmetric but not positive definite (its eigenvalues are 1 and -1).

2. Step 2: Understand cg requirements

   The conjugate gradient method requires A to be symmetric positive definite. Since A is not, cg will fail or not converge properly.

3. Final Answer:

   Matrix A is not symmetric positive definite, so cg will fail. -> Option A

4. Quick Check:

   cg needs symmetric positive definite matrix [OK]

Hint: Check matrix eigenvalues before using cg [OK]

Common Mistakes:

• Ignoring matrix definiteness
• Assuming cg works for any symmetric matrix
• Thinking vector shape causes error

5. You have a large sparse matrix that is not symmetric positive definite. Which solver should you use to solve Ax = b efficiently?

hard

A. Use cg because it is always faster.

B. Convert matrix to dense and use direct solver.

C. Use gmres because it works for general matrices.

D. Use cg after transposing the matrix.

Solution

1. Step 1: Identify matrix properties

   The matrix is large, sparse, and not symmetric positive definite, so cg is not suitable.

2. Step 2: Choose appropriate solver

   gmres can handle general matrices efficiently without requiring symmetry or positive definiteness.

3. Step 3: Avoid dense conversion

   Converting to dense wastes memory and time, so it is not efficient.

4. Final Answer:

   Use gmres because it works for general matrices. -> Option C

5. Quick Check:

   gmres handles general sparse matrices [OK]

Hint: Use gmres for non-symmetric or indefinite sparse matrices [OK]

Common Mistakes:

• Trying to use cg on non-symmetric matrices
• Converting sparse to dense unnecessarily
• Thinking transposing fixes definiteness