Spearman correlation in SciPy - Step-by-Step Execution

Concept Flow - Spearman correlation

Start with two data lists

↓

Rank each list separately

↓

Calculate difference of ranks

↓

Compute Spearman correlation coefficient

↓

Output correlation and p-value

↓

End

Spearman correlation measures how well two sets of data relate by comparing their ranks, not the raw values.

Execution Sample

SciPy

from scipy.stats import spearmanr
x = [10, 20, 30, 40, 50]
y = [12, 24, 33, 48, 55]
correlation, pvalue = spearmanr(x, y)
print(correlation, pvalue)

Calculate Spearman correlation between two lists of numbers and print the correlation coefficient and p-value.

Execution Table

Step	Action	x ranks	y ranks	Rank differences (d)	d^2	Correlation	p-value
1	Input data	[10, 20, 30, 40, 50]	[12, 24, 33, 48, 55]
2	Rank x	[1, 2, 3, 4, 5]
3	Rank y		[1, 2, 3, 4, 5]
4	Calculate d = x_rank - y_rank			[0, 0, 0, 0, 0]
5	Calculate d^2				[0, 0, 0, 0, 0]
6	Compute Spearman correlation					1.0	1.4042654220543672e-24
7	Output result					1.0	1.4042654220543672e-24

💡 All ranks match perfectly, so correlation is 1.0 and p-value is very small indicating strong correlation.

Variable Tracker

Variable	Start	After Step 2	After Step 3	After Step 4	After Step 5	Final
x	[10, 20, 30, 40, 50]	[1, 2, 3, 4, 5]	[1, 2, 3, 4, 5]	[1, 2, 3, 4, 5]	[1, 2, 3, 4, 5]	[1, 2, 3, 4, 5]
y	[12, 24, 33, 48, 55]	[12, 24, 33, 48, 55]	[1, 2, 3, 4, 5]	[1, 2, 3, 4, 5]	[1, 2, 3, 4, 5]	[1, 2, 3, 4, 5]
d				[0, 0, 0, 0, 0]	[0, 0, 0, 0, 0]	[0, 0, 0, 0, 0]
d^2					[0, 0, 0, 0, 0]	[0, 0, 0, 0, 0]
correlation						1.0
p-value						1.4042654220543672e-24

Key Moments - 3 Insights

Why do we rank the data instead of using the raw values?

Spearman correlation measures how well the order of data matches, not the exact values. Ranking converts values to their order positions, which is why in execution_table rows 2 and 3 we rank x and y.

What does a correlation of 1.0 mean here?

A correlation of 1.0 means the ranks match perfectly with no differences (d=0 for all), as shown in execution_table rows 4 and 5 where all d and d^2 are zero.

Why is the p-value so small?

The p-value shows the chance of seeing this correlation by random chance. Since correlation is perfect, p-value is very small, indicating strong evidence of correlation (execution_table row 6).

Visual Quiz - 3 Questions

Look at the execution table at step 4. What are the rank differences (d) between x and y?

A [0, 0, 0, 0, 0]

B [1, 1, 1, 1, 1]

C [-1, -1, -1, -1, -1]

D [5, 4, 3, 2, 1]

💡 Hint

Check the 'Rank differences (d)' column in execution_table row 4.

At which step does the code calculate the Spearman correlation coefficient?

A Step 3

B Step 5

C Step 6

D Step 7

💡 Hint

Look at the 'Correlation' column in execution_table to find when the value appears.

If the y data were reversed, how would the correlation value change in the table?

A It would stay 1.0

B It would become -1.0

C It would become 0

D It would become a random value

💡 Hint

Spearman correlation measures monotonic relationship; reversing order flips sign.

Concept Snapshot

Spearman correlation measures how well two variables relate by comparing their ranks.
Use scipy.stats.spearmanr(x, y) to get correlation and p-value.
Ranks replace raw values to focus on order, not magnitude.
Correlation ranges from -1 (perfect opposite) to 1 (perfect match).
P-value shows significance of the correlation.

Full Transcript

Spearman correlation is a way to measure how two sets of data relate by looking at their order, not their exact values. We start with two lists of numbers. Then, we rank each list separately, turning the numbers into their positions in order. Next, we find the difference between these ranks for each pair. We square these differences and use them to calculate the Spearman correlation coefficient. This coefficient tells us how closely the two lists match in order. A value of 1 means perfect match, -1 means perfect opposite, and 0 means no relation. The p-value tells us if this correlation is likely due to chance. In our example, the ranks match perfectly, so the correlation is 1 and the p-value is very small, showing a strong relationship.