Correlation analysis (Pearson, Spearman) in Data Analysis Python - Step-by-Step Execution

Concept Flow - Correlation analysis (Pearson, Spearman)

Start with two data sets

↓

Choose correlation type

↓

Pearson

↓

Calculate correlation coefficient

↓

Interpret strength and direction

↓

End

We start with two sets of data, choose Pearson or Spearman correlation, calculate the coefficient, then interpret the result.

Execution Sample

Data Analysis Python

from scipy.stats import pearsonr, spearmanr

x = [1, 2, 3, 4, 5]
y = [5, 6, 7, 8, 9]

pearson_corr, _ = pearsonr(x, y)
spearman_corr, _ = spearmanr(x, y)

print(pearson_corr, spearman_corr)

This code calculates Pearson and Spearman correlation coefficients between two lists of numbers.

Execution Table

Step	Action	Input Data	Calculation	Result
1	Prepare data	x=[1,2,3,4,5], y=[5,6,7,8,9]	None	Data ready
2	Calculate Pearson correlation	x, y	Compute covariance and standard deviations	Pearson r = 1.0
3	Calculate Spearman correlation	x, y	Rank data and compute Pearson on ranks	Spearman rho = 1.0
4	Print results	Pearson r=1.0, Spearman rho=1.0	Output values	1.0 1.0
5	End	None	None	Execution complete

💡 All calculations done, results printed, execution ends

Variable Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	Final
x	None	[1,2,3,4,5]	[1,2,3,4,5]	[1,2,3,4,5]	[1,2,3,4,5]
y	None	[5,6,7,8,9]	[5,6,7,8,9]	[5,6,7,8,9]	[5,6,7,8,9]
pearson_corr	None	None	1.0	1.0	1.0
spearman_corr	None	None	None	1.0	1.0

Key Moments - 3 Insights

Why do Pearson and Spearman correlations give the same result here?

Because the data is already in a monotonic increasing order with no ties, ranking the data (Spearman) does not change the relationship, so both coefficients are equal as shown in steps 2 and 3.

What does a correlation coefficient of 1.0 mean?

It means a perfect positive relationship between x and y; as x increases, y tends to increase too, as seen in the results in step 4.

Why do we calculate Spearman correlation by ranking data?

Spearman correlation measures monotonic relationships by ranking data first, which makes it less sensitive to outliers and non-linear relationships, as explained in step 3.

Visual Quiz - 3 Questions

Look at the execution table, what is the Pearson correlation coefficient after step 2?

A 1.0

B 0.5

C -0.9

D 0.9

💡 Hint

Check the 'Result' column in row for step 2 in the execution_table.

At which step do we rank the data to calculate Spearman correlation?

A Step 1

B Step 2

C Step 3

D Step 4

💡 Hint

Look at the 'Calculation' column for step 3 in the execution_table.

If the data had many tied ranks, which correlation would be more reliable?

A Pearson

B Spearman

C Both equal

D Neither

💡 Hint

Refer to the key_moments explanation about ranking and sensitivity to ties.

Concept Snapshot

Correlation analysis measures how two variables move together.
Pearson correlation measures linear relationships.
Spearman correlation measures monotonic relationships using ranks.
Values range from -1 (perfect negative) to 1 (perfect positive).
Use Pearson for linear data, Spearman for non-linear or ranked data.

Full Transcript

Correlation analysis compares two data sets to find how they relate. Pearson correlation checks if they change together in a straight line. Spearman correlation checks if one goes up when the other goes up, even if not straight. We start with data, pick Pearson or Spearman, calculate the number, then see if it's close to 1 or -1 to know if they are strongly related. In the example, both correlations are 1.0, showing a strong positive link.

Practice

1. What does the Pearson correlation coefficient measure between two variables?

easy

A. The rank order similarity between two variables

B. The difference in means between two groups

C. The strength and direction of a linear relationship

D. The frequency distribution of a variable

Solution

1. Step 1: Understand Pearson correlation concept

   Pearson correlation measures how strongly two variables move together in a straight line.

2. Step 2: Differentiate from other measures

   Unlike Spearman, Pearson focuses on linear relationships, not ranks or means.

3. Final Answer:

   The strength and direction of a linear relationship -> Option C

4. Quick Check:

   Pearson = linear relationship [OK]

Hint: Pearson = linear, Spearman = rank-based [OK]

Common Mistakes:

• Confusing Pearson with Spearman correlation
• Thinking Pearson measures differences in means
• Assuming Pearson works for non-linear relationships

2. Which of the following Python code snippets correctly calculates the Spearman correlation between two pandas Series x and y?

easy

A. x.corr(y, method='spearman')

B. x.corr(y, method='pearson')

C. spearmanr(x, y)

D. corr(x, y, method='spearman')

Solution

1. Step 1: Recall pandas correlation syntax

   In pandas, Series.corr() accepts a method argument with values like 'pearson' or 'spearman'.

2. Step 2: Identify correct method call

   x.corr(y, method='spearman') uses x.corr(y, method='spearman'), which is the correct pandas syntax.

3. Final Answer:

   x.corr(y, method='spearman') -> Option A

4. Quick Check:

   pandas Series.corr with method='spearman' [OK]

Hint: Use Series.corr with method='spearman' for Spearman correlation [OK]

Common Mistakes:

• Using 'pearson' instead of 'spearman' for Spearman correlation
• Calling a non-existent function like corr() standalone
• Confusing scipy's spearmanr with pandas syntax

3. Given the following code, what is the output of print(round(corr, 2))?

import pandas as pd
x = pd.Series([1, 2, 3, 4, 5])
y = pd.Series([5, 6, 7, 8, 8])
corr = x.corr(y, method='pearson')

medium

A. 0.75

B. 0.90

C. 0.85

D. 0.97

Solution

1. Step 1: Calculate Pearson correlation manually

   Calculate covariance and standard deviations of x and y, then compute correlation. Here, x and y increase mostly together, so correlation is high.

2. Step 2: Use pandas to confirm

   Running the code gives approximately 0.9746, which rounds to 0.97.

3. Final Answer:

   0.97 -> Option D

4. Quick Check:

   pandas Pearson correlation ≈ 0.97 [OK]

Hint: Use pandas corr() to get exact correlation quickly [OK]

Common Mistakes:

• Confusing Pearson with Spearman correlation values
• Rounding incorrectly or too early
• Assuming correlation must be exactly 1 for similar sequences

4. Identify the error in this code snippet that attempts to calculate Spearman correlation:

from scipy.stats import spearmanr
x = [1, 2, 3, 4]
y = [4, 3, 2, 1]
result = spearmanr(x, y)
print(result.correlation)

medium

A. The function spearmanr requires numpy arrays, not lists

B. The attribute 'correlation' does not exist in the result object

C. The print statement should be inside a function

D. The import statement is incorrect

Solution

1. Step 1: Check spearmanr return type

   spearmanr returns a namedtuple SpearmanRResult with fields statistic (Spearman correlation coefficient) and pvalue.

2. Step 2: Confirm attribute access

   The code accesses result.correlation, but no such attribute exists. Use result.statistic instead.

3. Final Answer:

   The attribute 'correlation' does not exist in the result object -> Option B

4. Quick Check:

   spearmanr uses result.statistic [OK]

Hint: Use result.statistic for spearmanr correlation [OK]

Common Mistakes:

• Assuming lists cannot be used with spearmanr
• Misunderstanding the return type of spearmanr
• Incorrect import or syntax errors

5. You have two variables: age (numeric) and education_level (ordinal ranks: 1=High School, 2=Bachelor, 3=Master, 4=PhD). Which correlation method is most appropriate to measure their association and why?

hard

A. Spearman, because education_level is ordinal and relationship may be monotonic

B. Pearson, because it handles ordinal data better

C. Neither, because correlation cannot be calculated between these variables

D. Pearson, because both are numeric variables

Solution

1. Step 1: Identify variable types

   Age is numeric continuous; education_level is ordinal categorical (ranked).

2. Step 2: Choose correlation method

   Spearman correlation is best for ordinal data and monotonic relationships, unlike Pearson which assumes linear and interval data.

3. Final Answer:

   Spearman, because education_level is ordinal and relationship may be monotonic -> Option A

4. Quick Check:

   Ordinal data + monotonic = Spearman [OK]

Hint: Use Spearman for ordinal or ranked data [OK]

Common Mistakes:

• Using Pearson for ordinal data incorrectly
• Thinking correlation can't be done with ordinal variables
• Confusing monotonic with linear relationships