Why Correlation analysis (Pearson, Spearman) in Data Analysis Python? - Purpose & Use Cases

The Big Idea

What if you could instantly know how two things truly relate without guessing or slow counting?

The Scenario

Imagine you have a big list of numbers from two different things, like hours studied and test scores, and you want to know if they are connected. Doing this by hand means checking every pair and guessing if one goes up when the other does.

The Problem

Doing this manually is slow and confusing. You might miss patterns or make mistakes counting. It's hard to tell if the connection is strong or just a coincidence. This makes it tough to trust your results.

The Solution

Correlation analysis uses simple math formulas to quickly and correctly measure how two things move together. Pearson checks straight-line connections, while Spearman looks at ranked order, helping you find hidden links easily and accurately.

Before vs After

✗ Before

count = 0
for i in range(1, len(x)):
    if x[i] > x[i-1] and y[i] > y[i-1]:
        count += 1
print(count)

✓ After

import scipy.stats as stats
pearson_corr, _ = stats.pearsonr(x, y)
spearman_corr, _ = stats.spearmanr(x, y)
print(pearson_corr, spearman_corr)

What It Enables

It lets you quickly find and trust the strength and type of relationships between data, helping you make smarter decisions.

Real Life Example

A teacher uses correlation analysis to see if more study hours really lead to better test scores, helping improve teaching methods.

Key Takeaways

Manual checking of relationships is slow and error-prone.

Correlation analysis gives fast, reliable measures of connection.

Pearson and Spearman cover different types of relationships.

Practice

1. What does the Pearson correlation coefficient measure between two variables?

easy

A. The rank order similarity between two variables

B. The difference in means between two groups

C. The strength and direction of a linear relationship

D. The frequency distribution of a variable

Solution

1. Step 1: Understand Pearson correlation concept

   Pearson correlation measures how strongly two variables move together in a straight line.

2. Step 2: Differentiate from other measures

   Unlike Spearman, Pearson focuses on linear relationships, not ranks or means.

3. Final Answer:

   The strength and direction of a linear relationship -> Option C

4. Quick Check:

   Pearson = linear relationship [OK]

Hint: Pearson = linear, Spearman = rank-based [OK]

Common Mistakes:

• Confusing Pearson with Spearman correlation
• Thinking Pearson measures differences in means
• Assuming Pearson works for non-linear relationships

2. Which of the following Python code snippets correctly calculates the Spearman correlation between two pandas Series x and y?

easy

A. x.corr(y, method='spearman')

B. x.corr(y, method='pearson')

C. spearmanr(x, y)

D. corr(x, y, method='spearman')

Solution

1. Step 1: Recall pandas correlation syntax

   In pandas, Series.corr() accepts a method argument with values like 'pearson' or 'spearman'.

2. Step 2: Identify correct method call

   x.corr(y, method='spearman') uses x.corr(y, method='spearman'), which is the correct pandas syntax.

3. Final Answer:

   x.corr(y, method='spearman') -> Option A

4. Quick Check:

   pandas Series.corr with method='spearman' [OK]

Hint: Use Series.corr with method='spearman' for Spearman correlation [OK]

Common Mistakes:

• Using 'pearson' instead of 'spearman' for Spearman correlation
• Calling a non-existent function like corr() standalone
• Confusing scipy's spearmanr with pandas syntax

3. Given the following code, what is the output of print(round(corr, 2))?

import pandas as pd
x = pd.Series([1, 2, 3, 4, 5])
y = pd.Series([5, 6, 7, 8, 8])
corr = x.corr(y, method='pearson')

medium

A. 0.75

B. 0.90

C. 0.85

D. 0.97

Solution

1. Step 1: Calculate Pearson correlation manually

   Calculate covariance and standard deviations of x and y, then compute correlation. Here, x and y increase mostly together, so correlation is high.

2. Step 2: Use pandas to confirm

   Running the code gives approximately 0.9746, which rounds to 0.97.

3. Final Answer:

   0.97 -> Option D

4. Quick Check:

   pandas Pearson correlation ≈ 0.97 [OK]

Hint: Use pandas corr() to get exact correlation quickly [OK]

Common Mistakes:

• Confusing Pearson with Spearman correlation values
• Rounding incorrectly or too early
• Assuming correlation must be exactly 1 for similar sequences

4. Identify the error in this code snippet that attempts to calculate Spearman correlation:

from scipy.stats import spearmanr
x = [1, 2, 3, 4]
y = [4, 3, 2, 1]
result = spearmanr(x, y)
print(result.correlation)

medium

A. The function spearmanr requires numpy arrays, not lists

B. The attribute 'correlation' does not exist in the result object

C. The print statement should be inside a function

D. The import statement is incorrect

Solution

1. Step 1: Check spearmanr return type

   spearmanr returns a namedtuple SpearmanRResult with fields statistic (Spearman correlation coefficient) and pvalue.

2. Step 2: Confirm attribute access

   The code accesses result.correlation, but no such attribute exists. Use result.statistic instead.

3. Final Answer:

   The attribute 'correlation' does not exist in the result object -> Option B

4. Quick Check:

   spearmanr uses result.statistic [OK]

Hint: Use result.statistic for spearmanr correlation [OK]

Common Mistakes:

• Assuming lists cannot be used with spearmanr
• Misunderstanding the return type of spearmanr
• Incorrect import or syntax errors

5. You have two variables: age (numeric) and education_level (ordinal ranks: 1=High School, 2=Bachelor, 3=Master, 4=PhD). Which correlation method is most appropriate to measure their association and why?

hard

A. Spearman, because education_level is ordinal and relationship may be monotonic

B. Pearson, because it handles ordinal data better

C. Neither, because correlation cannot be calculated between these variables

D. Pearson, because both are numeric variables

Solution

1. Step 1: Identify variable types

   Age is numeric continuous; education_level is ordinal categorical (ranked).

2. Step 2: Choose correlation method

   Spearman correlation is best for ordinal data and monotonic relationships, unlike Pearson which assumes linear and interval data.

3. Final Answer:

   Spearman, because education_level is ordinal and relationship may be monotonic -> Option A

4. Quick Check:

   Ordinal data + monotonic = Spearman [OK]

Hint: Use Spearman for ordinal or ranked data [OK]

Common Mistakes:

• Using Pearson for ordinal data incorrectly
• Thinking correlation can't be done with ordinal variables
• Confusing monotonic with linear relationships