Pearson correlation in SciPy - Step-by-Step Execution

Concept Flow - Pearson correlation

Start with two data lists

↓

Calculate means of both lists

↓

Calculate deviations from means

↓

Multiply deviations pairwise

↓

Sum products and calculate covariance

↓

Calculate standard deviations

↓

Divide covariance by product of std devs

↓

Output Pearson correlation coefficient

Pearson correlation measures how two sets of numbers move together, from -1 (opposite) to 1 (same).

Execution Sample

SciPy

from scipy.stats import pearsonr

x = [1, 2, 3, 4, 5]
y = [2, 4, 6, 8, 10]

corr, p_value = pearsonr(x, y)
print(corr)

Calculate Pearson correlation between two lists x and y.

Execution Table

Step	Action	Intermediate Values	Result
1	Calculate mean of x	mean_x = (1+2+3+4+5)/5 = 3.0	mean_x = 3.0
2	Calculate mean of y	mean_y = (2+4+6+8+10)/5 = 6.0	mean_y = 6.0
3	Calculate deviations x - mean_x	[-2, -1, 0, 1, 2]	deviations_x = [-2, -1, 0, 1, 2]
4	Calculate deviations y - mean_y	[-4, -2, 0, 2, 4]	deviations_y = [-4, -2, 0, 2, 4]
5	Multiply deviations pairwise	[-2*-4, -1*-2, 0*0, 1*2, 2*4] = [8, 2, 0, 2, 8]	products = [8, 2, 0, 2, 8]
6	Sum products	8 + 2 + 0 + 2 + 8 = 20	sum_products = 20
7	Calculate covariance	cov = sum_products / (5 - 1) = 20 / 4 = 5.0	covariance = 5.0
8	Calculate std dev of x	sqrt(sum((x_i - mean_x)^2) / (5 - 1)) = sqrt(10 / 4) = 1.5811	std_x = 1.5811
9	Calculate std dev of y	sqrt(sum((y_i - mean_y)^2) / (5 - 1)) = sqrt(40 / 4) = 3.1623	std_y = 3.1623
10	Calculate Pearson correlation	covariance / (std_x * std_y) = 5.0 / (1.5811 * 3.1623) = 1.0	correlation = 1.0
11	Output correlation	correlation = 1.0	1.0

💡 All data processed, Pearson correlation calculated as 1.0

Variable Tracker

Variable	Start	After Step 3	After Step 4	After Step 5	After Step 6	After Step 7	After Step 8	After Step 9	After Step 10	Final
x	[1,2,3,4,5]	[-2, -1, 0, 1, 2]	[-2, -1, 0, 1, 2]	[-2, -1, 0, 1, 2]	[-2, -1, 0, 1, 2]	[-2, -1, 0, 1, 2]	[-2, -1, 0, 1, 2]	[-2, -1, 0, 1, 2]	[-2, -1, 0, 1, 2]	[-2, -1, 0, 1, 2]
y	[2,4,6,8,10]	[2,4,6,8,10]	[-4, -2, 0, 2, 4]	[-4, -2, 0, 2, 4]	[-4, -2, 0, 2, 4]	[-4, -2, 0, 2, 4]	[-4, -2, 0, 2, 4]	[-4, -2, 0, 2, 4]	[-4, -2, 0, 2, 4]	[-4, -2, 0, 2, 4]
products	N/A	N/A	N/A	[8, 2, 0, 2, 8]	[8, 2, 0, 2, 8]	[8, 2, 0, 2, 8]	[8, 2, 0, 2, 8]	[8, 2, 0, 2, 8]	[8, 2, 0, 2, 8]	[8, 2, 0, 2, 8]
sum_products	N/A	N/A	N/A	N/A	20	20	20	20	20	20
covariance	N/A	N/A	N/A	N/A	N/A	5.0	5.0	5.0	5.0	5.0
std_x	N/A	N/A	N/A	N/A	N/A	N/A	1.5811	1.5811	1.5811	1.5811
std_y	N/A	N/A	N/A	N/A	N/A	N/A	N/A	3.1623	3.1623	3.1623
correlation	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	1.0	1.0

Key Moments - 3 Insights

Why do we divide by (n-1) instead of n when calculating covariance and standard deviation?

Dividing by (n-1) gives an unbiased estimate of variance and covariance for sample data, as shown in steps 7, 8, and 9 of the execution_table.

Why is the Pearson correlation exactly 1.0 in this example?

Because y is exactly twice x, their deviations move perfectly together, making the correlation 1.0 as shown in step 10.

What do the deviations represent in steps 3 and 4?

Deviations show how far each value is from the mean, which helps measure how x and y move relative to their averages.

Visual Quiz - 3 Questions

Look at the execution_table at step 5, what is the product of deviations for the third pair (x=3, y=6)?

A 3

B 0

C 6

D -3

💡 Hint

Check the 'Multiply deviations pairwise' row in execution_table step 5.

At which step does the calculation of standard deviation of y happen?

A Step 9

B Step 8

C Step 7

D Step 10

💡 Hint

Look for 'Calculate std dev of y' in execution_table.

If the values in y were all the same, how would the Pearson correlation change?

A It would be 1.0

B It would be 0

C It would be undefined or cause an error

D -1.0

💡 Hint

Recall that standard deviation of y would be zero, causing division by zero in step 10.

Concept Snapshot

Pearson correlation measures linear relationship between two numeric lists.
Use scipy.stats.pearsonr(x, y) to get correlation and p-value.
Returns value between -1 (opposite) and 1 (same direction).
Calculated as covariance divided by product of standard deviations.
Requires at least two data points in each list.

Full Transcript

Pearson correlation shows how two sets of numbers move together. We start by finding the average of each list. Then, for each number, we find how far it is from the average. We multiply these differences pair by pair and add them up. This sum helps us find covariance. We also find how spread out each list is using standard deviation. Finally, we divide covariance by the product of standard deviations to get the correlation number. This number tells us if the lists move together (close to 1), opposite (close to -1), or not related (close to 0).

Practice

1. What does the Pearson correlation coefficient measure between two variables?

easy

A. The frequency of values in each variable

B. The difference in means between two groups

C. The total sum of values in each variable

D. The strength and direction of a linear relationship

Solution

1. Step 1: Understand Pearson correlation purpose

   Pearson correlation measures how two variables move together in a straight line.

2. Step 2: Identify what it quantifies

   It quantifies both strength (how close points are to a line) and direction (positive or negative slope).

3. Final Answer:

   The strength and direction of a linear relationship -> Option D

4. Quick Check:

   Pearson correlation = strength and direction [OK]

Hint: Pearson = linear relationship strength and direction [OK]

Common Mistakes:

• Confusing correlation with difference in means
• Thinking it measures frequency counts
• Assuming it measures total sums

2. Which of the following is the correct way to import the Pearson correlation function from scipy?

easy

A. from scipy.stats import pearsonr

B. import scipy.pearsonr

C. from scipy import pearsonr

D. import pearsonr from scipy.stats

Solution

1. Step 1: Recall scipy.stats module structure

   The pearsonr function is inside scipy.stats, so import must reflect that.

2. Step 2: Check correct import syntax

   Python import syntax for functions is 'from module import function', so 'from scipy.stats import pearsonr' is correct.

3. Final Answer:

   from scipy.stats import pearsonr -> Option A

4. Quick Check:

   Correct import syntax = from scipy.stats import pearsonr [OK]

Hint: Use 'from scipy.stats import pearsonr' to import [OK]

Common Mistakes:

• Using 'import scipy.pearsonr' which is invalid
• Trying 'from scipy import pearsonr' missing stats
• Wrong syntax like 'import pearsonr from scipy.stats'

3. What is the output of this code?

from scipy.stats import pearsonr
x = [1, 2, 3, 4, 5]
y = [2, 4, 6, 8, 10]
corr, p = pearsonr(x, y)
print("{:.2f}".format(corr))

medium

A. 0.00

B. -1.00

C. 1.00

D. 0.50

Solution

1. Step 1: Analyze the data relationship

   y is exactly 2 times x, so they have a perfect positive linear relationship.

2. Step 2: Understand Pearson correlation output

   Perfect positive linear relationship means correlation coefficient is 1.0.

3. Final Answer:

   1.00 -> Option C

4. Quick Check:

   Perfect positive linear correlation = 1.00 [OK]

Hint: Perfect linear scaling means correlation = 1.00 [OK]

Common Mistakes:

• Confusing correlation with p-value
• Expecting 0.5 for proportional data
• Ignoring perfect linear relationship

4. Identify the error in this code snippet:

from scipy.stats import pearsonr
x = [1, 2, 3]
y = [4, 5]
corr, p = pearsonr(x, y)
print(corr)

medium

A. Lists x and y have different lengths

B. pearsonr function is not imported correctly

C. Missing parentheses in print statement

D. Variables x and y must be numpy arrays

Solution

1. Step 1: Check input data lengths

   pearsonr requires x and y to have the same length. Here, x has 3 elements, y has 2.

2. Step 2: Understand error cause

   Different lengths cause a ValueError when calling pearsonr.

3. Final Answer:

   Lists x and y have different lengths -> Option A

4. Quick Check:

   Input length mismatch causes error [OK]

Hint: Ensure x and y lists have equal length [OK]

Common Mistakes:

• Assuming pearsonr accepts different length inputs
• Thinking print syntax is wrong in Python 3
• Believing inputs must be numpy arrays

5. You have two variables: temperature (in Celsius) and ice cream sales. You want to check if higher temperature relates to higher sales. Which code correctly computes the Pearson correlation and interprets a strong positive relationship?

hard

A. from scipy.stats import pearsonr
corr, p = pearsonr(temp, sales)
if p > 0.7:
print('Strong positive correlation')

B. from scipy.stats import pearsonr
corr, p = pearsonr(temp, sales)
if corr > 0.7:
print('Strong positive correlation')

C. from scipy.stats import pearsonr
corr, p = pearsonr(temp, sales)
if corr < -0.7:
print('Strong positive correlation')

D. from scipy.stats import pearsonr
corr, p = pearsonr(temp, sales)
if abs(corr) < 0.3:
print('Strong positive correlation')

Solution

1. Step 1: Calculate Pearson correlation

   Use pearsonr to get correlation coefficient and p-value between temperature and sales.

2. Step 2: Interpret correlation coefficient

   A strong positive correlation means corr > 0.7. Checking p-value > 0.7 or negative corr is incorrect.

3. Final Answer:

   from scipy.stats import pearsonr corr, p = pearsonr(temp, sales) if corr > 0.7: print('Strong positive correlation') -> Option B

4. Quick Check:

   Strong positive correlation if corr > 0.7 [OK]

Hint: Check corr > 0.7 for strong positive relation [OK]

Common Mistakes:

• Using p-value instead of correlation coefficient
• Checking for negative correlation for positive relation
• Using absolute value less than threshold incorrectly