Correlation analysis (Pearson, Spearman) in Data Analysis Python

Introduction

Correlation analysis helps us find out how two things change together. It tells if one thing goes up when the other goes up or down.

To check if study time and exam scores are related.

To see if temperature and ice cream sales move together.

To find if height and weight have a connection.

To understand if advertising budget affects product sales.

Syntax

Data Analysis Python

from scipy.stats import pearsonr, spearmanr

# For Pearson correlation
correlation, p_value = pearsonr(data1, data2)

# For Spearman correlation
correlation, p_value = spearmanr(data1, data2)

pearsonr measures linear relationship between two numeric sets.

spearmanr measures monotonic relationship using ranks, good for non-linear or ordinal data.

Examples

Calculates Pearson correlation for two lists with perfect positive linear relation.

Data Analysis Python

from scipy.stats import pearsonr

x = [1, 2, 3, 4, 5]
y = [2, 4, 6, 8, 10]
correlation, p_value = pearsonr(x, y)
print(correlation)

Calculates Spearman correlation for data that may not be perfectly linear but has a monotonic trend.

Data Analysis Python

from scipy.stats import spearmanr

x = [1, 2, 3, 4, 5]
y = [5, 6, 7, 8, 7]
correlation, p_value = spearmanr(x, y)
print(correlation)

Sample Program

This program calculates both Pearson and Spearman correlations between hours studied and exam scores to see how strongly they are related.

Data Analysis Python

from scipy.stats import pearsonr, spearmanr

# Sample data: hours studied and exam scores
hours_studied = [1, 2, 3, 4, 5, 6, 7, 8]
exam_scores = [50, 55, 65, 70, 75, 80, 85, 90]

# Calculate Pearson correlation
pearson_corr, pearson_p = pearsonr(hours_studied, exam_scores)

# Calculate Spearman correlation
spearman_corr, spearman_p = spearmanr(hours_studied, exam_scores)

print(f"Pearson correlation: {pearson_corr:.2f}")
print(f"Spearman correlation: {spearman_corr:.2f}")

Important Notes

Correlation values range from -1 to 1. Close to 1 means strong positive relation, close to -1 means strong negative relation, and near 0 means no relation.

Use Pearson when data is linear and numeric. Use Spearman when data is not linear or has ranks.

Always check p-value to see if correlation is statistically significant (usually p < 0.05).

Summary

Correlation analysis shows how two variables move together.

Pearson measures linear relationships; Spearman measures monotonic relationships.

Both give a correlation value and a p-value to check significance.

Practice

1. What does the Pearson correlation coefficient measure between two variables?

easy

A. The rank order similarity between two variables

B. The difference in means between two groups

C. The strength and direction of a linear relationship

D. The frequency distribution of a variable

Solution

1. Step 1: Understand Pearson correlation concept

   Pearson correlation measures how strongly two variables move together in a straight line.

2. Step 2: Differentiate from other measures

   Unlike Spearman, Pearson focuses on linear relationships, not ranks or means.

3. Final Answer:

   The strength and direction of a linear relationship -> Option C

4. Quick Check:

   Pearson = linear relationship [OK]

Hint: Pearson = linear, Spearman = rank-based [OK]

Common Mistakes:

• Confusing Pearson with Spearman correlation
• Thinking Pearson measures differences in means
• Assuming Pearson works for non-linear relationships

2. Which of the following Python code snippets correctly calculates the Spearman correlation between two pandas Series x and y?

easy

A. x.corr(y, method='spearman')

B. x.corr(y, method='pearson')

C. spearmanr(x, y)

D. corr(x, y, method='spearman')

Solution

1. Step 1: Recall pandas correlation syntax

   In pandas, Series.corr() accepts a method argument with values like 'pearson' or 'spearman'.

2. Step 2: Identify correct method call

   x.corr(y, method='spearman') uses x.corr(y, method='spearman'), which is the correct pandas syntax.

3. Final Answer:

   x.corr(y, method='spearman') -> Option A

4. Quick Check:

   pandas Series.corr with method='spearman' [OK]

Hint: Use Series.corr with method='spearman' for Spearman correlation [OK]

Common Mistakes:

• Using 'pearson' instead of 'spearman' for Spearman correlation
• Calling a non-existent function like corr() standalone
• Confusing scipy's spearmanr with pandas syntax

3. Given the following code, what is the output of print(round(corr, 2))?

import pandas as pd
x = pd.Series([1, 2, 3, 4, 5])
y = pd.Series([5, 6, 7, 8, 8])
corr = x.corr(y, method='pearson')

medium

A. 0.75

B. 0.90

C. 0.85

D. 0.97

Solution

1. Step 1: Calculate Pearson correlation manually

   Calculate covariance and standard deviations of x and y, then compute correlation. Here, x and y increase mostly together, so correlation is high.

2. Step 2: Use pandas to confirm

   Running the code gives approximately 0.9746, which rounds to 0.97.

3. Final Answer:

   0.97 -> Option D

4. Quick Check:

   pandas Pearson correlation ≈ 0.97 [OK]

Hint: Use pandas corr() to get exact correlation quickly [OK]

Common Mistakes:

• Confusing Pearson with Spearman correlation values
• Rounding incorrectly or too early
• Assuming correlation must be exactly 1 for similar sequences

4. Identify the error in this code snippet that attempts to calculate Spearman correlation:

from scipy.stats import spearmanr
x = [1, 2, 3, 4]
y = [4, 3, 2, 1]
result = spearmanr(x, y)
print(result.correlation)

medium

A. The function spearmanr requires numpy arrays, not lists

B. The attribute 'correlation' does not exist in the result object

C. The print statement should be inside a function

D. The import statement is incorrect

Solution

1. Step 1: Check spearmanr return type

   spearmanr returns a namedtuple SpearmanRResult with fields statistic (Spearman correlation coefficient) and pvalue.

2. Step 2: Confirm attribute access

   The code accesses result.correlation, but no such attribute exists. Use result.statistic instead.

3. Final Answer:

   The attribute 'correlation' does not exist in the result object -> Option B

4. Quick Check:

   spearmanr uses result.statistic [OK]

Hint: Use result.statistic for spearmanr correlation [OK]

Common Mistakes:

• Assuming lists cannot be used with spearmanr
• Misunderstanding the return type of spearmanr
• Incorrect import or syntax errors

5. You have two variables: age (numeric) and education_level (ordinal ranks: 1=High School, 2=Bachelor, 3=Master, 4=PhD). Which correlation method is most appropriate to measure their association and why?

hard

A. Spearman, because education_level is ordinal and relationship may be monotonic

B. Pearson, because it handles ordinal data better

C. Neither, because correlation cannot be calculated between these variables

D. Pearson, because both are numeric variables

Solution

1. Step 1: Identify variable types

   Age is numeric continuous; education_level is ordinal categorical (ranked).

2. Step 2: Choose correlation method

   Spearman correlation is best for ordinal data and monotonic relationships, unlike Pearson which assumes linear and interval data.

3. Final Answer:

   Spearman, because education_level is ordinal and relationship may be monotonic -> Option A

4. Quick Check:

   Ordinal data + monotonic = Spearman [OK]

Hint: Use Spearman for ordinal or ranked data [OK]

Common Mistakes:

• Using Pearson for ordinal data incorrectly
• Thinking correlation can't be done with ordinal variables
• Confusing monotonic with linear relationships