Pearson correlation in SciPy - Step-by-Step Execution

Concept Flow - Pearson correlation

Start with two data lists

↓

Calculate means of both lists

↓

Calculate deviations from means

↓

Multiply deviations pairwise

↓

Sum products and calculate covariance

↓

Calculate standard deviations

↓

Divide covariance by product of std devs

↓

Output Pearson correlation coefficient

Pearson correlation measures how two sets of numbers move together, from -1 (opposite) to 1 (same).

Execution Sample

SciPy

from scipy.stats import pearsonr

x = [1, 2, 3, 4, 5]
y = [2, 4, 6, 8, 10]

corr, p_value = pearsonr(x, y)
print(corr)

Calculate Pearson correlation between two lists x and y.

Execution Table

Step	Action	Intermediate Values	Result
1	Calculate mean of x	mean_x = (1+2+3+4+5)/5 = 3.0	mean_x = 3.0
2	Calculate mean of y	mean_y = (2+4+6+8+10)/5 = 6.0	mean_y = 6.0
3	Calculate deviations x - mean_x	[-2, -1, 0, 1, 2]	deviations_x = [-2, -1, 0, 1, 2]
4	Calculate deviations y - mean_y	[-4, -2, 0, 2, 4]	deviations_y = [-4, -2, 0, 2, 4]
5	Multiply deviations pairwise	[-2*-4, -1*-2, 0*0, 1*2, 2*4] = [8, 2, 0, 2, 8]	products = [8, 2, 0, 2, 8]
6	Sum products	8 + 2 + 0 + 2 + 8 = 20	sum_products = 20
7	Calculate covariance	cov = sum_products / (5 - 1) = 20 / 4 = 5.0	covariance = 5.0
8	Calculate std dev of x	sqrt(sum((x_i - mean_x)^2) / (5 - 1)) = sqrt(10 / 4) = 1.5811	std_x = 1.5811
9	Calculate std dev of y	sqrt(sum((y_i - mean_y)^2) / (5 - 1)) = sqrt(40 / 4) = 3.1623	std_y = 3.1623
10	Calculate Pearson correlation	covariance / (std_x * std_y) = 5.0 / (1.5811 * 3.1623) = 1.0	correlation = 1.0
11	Output correlation	correlation = 1.0	1.0

💡 All data processed, Pearson correlation calculated as 1.0

Variable Tracker

Variable	Start	After Step 3	After Step 4	After Step 5	After Step 6	After Step 7	After Step 8	After Step 9	After Step 10	Final
x	[1,2,3,4,5]	[-2, -1, 0, 1, 2]	[-2, -1, 0, 1, 2]	[-2, -1, 0, 1, 2]	[-2, -1, 0, 1, 2]	[-2, -1, 0, 1, 2]	[-2, -1, 0, 1, 2]	[-2, -1, 0, 1, 2]	[-2, -1, 0, 1, 2]	[-2, -1, 0, 1, 2]
y	[2,4,6,8,10]	[2,4,6,8,10]	[-4, -2, 0, 2, 4]	[-4, -2, 0, 2, 4]	[-4, -2, 0, 2, 4]	[-4, -2, 0, 2, 4]	[-4, -2, 0, 2, 4]	[-4, -2, 0, 2, 4]	[-4, -2, 0, 2, 4]	[-4, -2, 0, 2, 4]
products	N/A	N/A	N/A	[8, 2, 0, 2, 8]	[8, 2, 0, 2, 8]	[8, 2, 0, 2, 8]	[8, 2, 0, 2, 8]	[8, 2, 0, 2, 8]	[8, 2, 0, 2, 8]	[8, 2, 0, 2, 8]
sum_products	N/A	N/A	N/A	N/A	20	20	20	20	20	20
covariance	N/A	N/A	N/A	N/A	N/A	5.0	5.0	5.0	5.0	5.0
std_x	N/A	N/A	N/A	N/A	N/A	N/A	1.5811	1.5811	1.5811	1.5811
std_y	N/A	N/A	N/A	N/A	N/A	N/A	N/A	3.1623	3.1623	3.1623
correlation	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	1.0	1.0

Key Moments - 3 Insights

Why do we divide by (n-1) instead of n when calculating covariance and standard deviation?

Dividing by (n-1) gives an unbiased estimate of variance and covariance for sample data, as shown in steps 7, 8, and 9 of the execution_table.

Why is the Pearson correlation exactly 1.0 in this example?

Because y is exactly twice x, their deviations move perfectly together, making the correlation 1.0 as shown in step 10.

What do the deviations represent in steps 3 and 4?

Deviations show how far each value is from the mean, which helps measure how x and y move relative to their averages.

Visual Quiz - 3 Questions

Look at the execution_table at step 5, what is the product of deviations for the third pair (x=3, y=6)?

A 3

B 0

C 6

D -3

💡 Hint

Check the 'Multiply deviations pairwise' row in execution_table step 5.

At which step does the calculation of standard deviation of y happen?

A Step 9

B Step 8

C Step 7

D Step 10

💡 Hint

Look for 'Calculate std dev of y' in execution_table.

If the values in y were all the same, how would the Pearson correlation change?

A It would be 1.0

B It would be 0

C It would be undefined or cause an error

D -1.0

💡 Hint

Recall that standard deviation of y would be zero, causing division by zero in step 10.

Concept Snapshot

Pearson correlation measures linear relationship between two numeric lists.
Use scipy.stats.pearsonr(x, y) to get correlation and p-value.
Returns value between -1 (opposite) and 1 (same direction).
Calculated as covariance divided by product of standard deviations.
Requires at least two data points in each list.

Full Transcript

Pearson correlation shows how two sets of numbers move together. We start by finding the average of each list. Then, for each number, we find how far it is from the average. We multiply these differences pair by pair and add them up. This sum helps us find covariance. We also find how spread out each list is using standard deviation. Finally, we divide covariance by the product of standard deviations to get the correlation number. This number tells us if the lists move together (close to 1), opposite (close to -1), or not related (close to 0).