Pearson correlation in SciPy

Introduction

Pearson correlation helps us see how two things change together. It tells if one goes up when the other goes up or down.

Checking if hours studied and exam scores are related.

Seeing if temperature and ice cream sales move together.

Finding if exercise time and weight loss are connected.

Understanding if advertising budget and product sales increase together.

Syntax

SciPy

from scipy.stats import pearsonr
correlation_coefficient, p_value = pearsonr(x, y)

x and y are lists or arrays of numbers with the same length.

The function returns two values: the correlation number and a p-value to check significance.

Examples

This shows a perfect positive correlation of 1.0 because y doubles x.

SciPy

from scipy.stats import pearsonr
x = [1, 2, 3, 4, 5]
y = [2, 4, 6, 8, 10]
corr, p = pearsonr(x, y)
print(corr)

This shows a perfect negative correlation of -1.0 because y decreases as x increases.

SciPy

from scipy.stats import pearsonr
x = [1, 2, 3, 4, 5]
y = [10, 8, 6, 4, 2]
corr, p = pearsonr(x, y)
print(corr)

This shows no correlation (nan) because y does not change at all.

SciPy

from scipy.stats import pearsonr
x = [1, 2, 3, 4, 5]
y = [5, 5, 5, 5, 5]
corr, p = pearsonr(x, y)
print(corr)

Sample Program

This program calculates how strongly hours studied and exam scores are related. It prints the correlation number and the p-value to check if the result is meaningful.

SciPy

from scipy.stats import pearsonr

# Example data: hours studied and exam scores
hours_studied = [1, 2, 3, 4, 5]
exam_scores = [50, 55, 65, 70, 80]

corr, p_value = pearsonr(hours_studied, exam_scores)

print(f"Pearson correlation coefficient: {corr:.2f}")
print(f"P-value: {p_value:.4f}")

Important Notes

The correlation value ranges from -1 to 1.

A value close to 1 means strong positive relation, close to -1 means strong negative relation, and around 0 means no relation.

The p-value helps decide if the correlation is likely real or by chance. A small p-value (like less than 0.05) means it is probably real.

Summary

Pearson correlation measures how two sets of numbers move together.

Use pearsonr from scipy.stats to get the correlation and p-value.

Values near 1 or -1 show strong relationships; near 0 means weak or no relationship.

Practice

1. What does the Pearson correlation coefficient measure between two variables?

easy

A. The frequency of values in each variable

B. The difference in means between two groups

C. The total sum of values in each variable

D. The strength and direction of a linear relationship

Solution

1. Step 1: Understand Pearson correlation purpose

   Pearson correlation measures how two variables move together in a straight line.

2. Step 2: Identify what it quantifies

   It quantifies both strength (how close points are to a line) and direction (positive or negative slope).

3. Final Answer:

   The strength and direction of a linear relationship -> Option D

4. Quick Check:

   Pearson correlation = strength and direction [OK]

Hint: Pearson = linear relationship strength and direction [OK]

Common Mistakes:

• Confusing correlation with difference in means
• Thinking it measures frequency counts
• Assuming it measures total sums

2. Which of the following is the correct way to import the Pearson correlation function from scipy?

easy

A. from scipy.stats import pearsonr

B. import scipy.pearsonr

C. from scipy import pearsonr

D. import pearsonr from scipy.stats

Solution

1. Step 1: Recall scipy.stats module structure

   The pearsonr function is inside scipy.stats, so import must reflect that.

2. Step 2: Check correct import syntax

   Python import syntax for functions is 'from module import function', so 'from scipy.stats import pearsonr' is correct.

3. Final Answer:

   from scipy.stats import pearsonr -> Option A

4. Quick Check:

   Correct import syntax = from scipy.stats import pearsonr [OK]

Hint: Use 'from scipy.stats import pearsonr' to import [OK]

Common Mistakes:

• Using 'import scipy.pearsonr' which is invalid
• Trying 'from scipy import pearsonr' missing stats
• Wrong syntax like 'import pearsonr from scipy.stats'

3. What is the output of this code?

from scipy.stats import pearsonr
x = [1, 2, 3, 4, 5]
y = [2, 4, 6, 8, 10]
corr, p = pearsonr(x, y)
print("{:.2f}".format(corr))

medium

A. 0.00

B. -1.00

C. 1.00

D. 0.50

Solution

1. Step 1: Analyze the data relationship

   y is exactly 2 times x, so they have a perfect positive linear relationship.

2. Step 2: Understand Pearson correlation output

   Perfect positive linear relationship means correlation coefficient is 1.0.

3. Final Answer:

   1.00 -> Option C

4. Quick Check:

   Perfect positive linear correlation = 1.00 [OK]

Hint: Perfect linear scaling means correlation = 1.00 [OK]

Common Mistakes:

• Confusing correlation with p-value
• Expecting 0.5 for proportional data
• Ignoring perfect linear relationship

4. Identify the error in this code snippet:

from scipy.stats import pearsonr
x = [1, 2, 3]
y = [4, 5]
corr, p = pearsonr(x, y)
print(corr)

medium

A. Lists x and y have different lengths

B. pearsonr function is not imported correctly

C. Missing parentheses in print statement

D. Variables x and y must be numpy arrays

Solution

1. Step 1: Check input data lengths

   pearsonr requires x and y to have the same length. Here, x has 3 elements, y has 2.

2. Step 2: Understand error cause

   Different lengths cause a ValueError when calling pearsonr.

3. Final Answer:

   Lists x and y have different lengths -> Option A

4. Quick Check:

   Input length mismatch causes error [OK]

Hint: Ensure x and y lists have equal length [OK]

Common Mistakes:

• Assuming pearsonr accepts different length inputs
• Thinking print syntax is wrong in Python 3
• Believing inputs must be numpy arrays

5. You have two variables: temperature (in Celsius) and ice cream sales. You want to check if higher temperature relates to higher sales. Which code correctly computes the Pearson correlation and interprets a strong positive relationship?

hard

A. from scipy.stats import pearsonr
corr, p = pearsonr(temp, sales)
if p > 0.7:
print('Strong positive correlation')

B. from scipy.stats import pearsonr
corr, p = pearsonr(temp, sales)
if corr > 0.7:
print('Strong positive correlation')

C. from scipy.stats import pearsonr
corr, p = pearsonr(temp, sales)
if corr < -0.7:
print('Strong positive correlation')

D. from scipy.stats import pearsonr
corr, p = pearsonr(temp, sales)
if abs(corr) < 0.3:
print('Strong positive correlation')

Solution

1. Step 1: Calculate Pearson correlation

   Use pearsonr to get correlation coefficient and p-value between temperature and sales.

2. Step 2: Interpret correlation coefficient

   A strong positive correlation means corr > 0.7. Checking p-value > 0.7 or negative corr is incorrect.

3. Final Answer:

   from scipy.stats import pearsonr corr, p = pearsonr(temp, sales) if corr > 0.7: print('Strong positive correlation') -> Option B

4. Quick Check:

   Strong positive correlation if corr > 0.7 [OK]

Hint: Check corr > 0.7 for strong positive relation [OK]

Common Mistakes:

• Using p-value instead of correlation coefficient
• Checking for negative correlation for positive relation
• Using absolute value less than threshold incorrectly