Correlation analysis (Pearson, Spearman) in Data Analysis Python - Mini Project: Build & Apply

Correlation analysis (Pearson, Spearman)

📖 Scenario: You are a data analyst working with a small dataset of students' study hours and their exam scores. You want to find out how strongly these two variables are related.

🎯 Goal: Build a Python program that calculates both Pearson and Spearman correlation coefficients between study hours and exam scores.

📋 What You'll Learn

Create a dictionary with exact study hours and exam scores

Create a list of tuples from the dictionary

Calculate Pearson and Spearman correlation coefficients using scipy.stats

Print the correlation results exactly as specified

💡 Why This Matters

🌍 Real World

Correlation analysis helps understand relationships between variables, like how study time affects exam scores.

💼 Career

Data analysts and scientists use correlation to find patterns and insights in data for decision making.

1

Create the dataset dictionary

Create a dictionary called student_data with these exact entries: 'hours': [2, 3, 5, 8, 13] and 'scores': [50, 55, 65, 80, 95].

Data Analysis Python

# Create the student_data dictionary with hours and scores
# Your code here

Hint

Use curly braces to create a dictionary with keys 'hours' and 'scores'. Assign the exact lists as values.

2

Prepare data for correlation

Create a list of tuples called data_pairs by pairing each element from student_data['hours'] with the corresponding element from student_data['scores'] using the zip function.

Data Analysis Python

student_data = {'hours': [2, 3, 5, 8, 13], 'scores': [50, 55, 65, 80, 95]}
# Create data_pairs as a list of tuples pairing hours and scores
# Your code here

Hint

Use zip(student_data['hours'], student_data['scores']) and convert it to a list.

3

Calculate Pearson and Spearman correlations

Import pearsonr and spearmanr from scipy.stats. Then calculate the Pearson correlation coefficient and Spearman correlation coefficient between student_data['hours'] and student_data['scores']. Store the coefficients in variables pearson_corr and spearman_corr respectively.

Data Analysis Python

student_data = {'hours': [2, 3, 5, 8, 13], 'scores': [50, 55, 65, 80, 95]}
data_pairs = list(zip(student_data['hours'], student_data['scores']))
# Import pearsonr and spearmanr from scipy.stats
# Calculate pearson_corr and spearman_corr
# Your code here

Hint

Use pearsonr(x, y) and spearmanr(x, y) to get correlation coefficients. Assign only the first returned value to variables.

4

Print the correlation results

Print the Pearson correlation coefficient with the exact text: "Pearson correlation: " followed by pearson_corr rounded to 3 decimal places. Then print the Spearman correlation coefficient with the exact text: "Spearman correlation: " followed by spearman_corr rounded to 3 decimal places.

Data Analysis Python

from scipy.stats import pearsonr, spearmanr

student_data = {'hours': [2, 3, 5, 8, 13], 'scores': [50, 55, 65, 80, 95]}
data_pairs = list(zip(student_data['hours'], student_data['scores']))

pearson_corr, _ = pearsonr(student_data['hours'], student_data['scores'])
spearman_corr, _ = spearmanr(student_data['hours'], student_data['scores'])
# Print the Pearson and Spearman correlation coefficients rounded to 3 decimals
# Your code here

Hint

Use print(f"Pearson correlation: {pearson_corr:.3f}") and similarly for Spearman.

Practice

1. What does the Pearson correlation coefficient measure between two variables?

easy

A. The rank order similarity between two variables

B. The difference in means between two groups

C. The strength and direction of a linear relationship

D. The frequency distribution of a variable

Solution

1. Step 1: Understand Pearson correlation concept

   Pearson correlation measures how strongly two variables move together in a straight line.

2. Step 2: Differentiate from other measures

   Unlike Spearman, Pearson focuses on linear relationships, not ranks or means.

3. Final Answer:

   The strength and direction of a linear relationship -> Option C

4. Quick Check:

   Pearson = linear relationship [OK]

Hint: Pearson = linear, Spearman = rank-based [OK]

Common Mistakes:

• Confusing Pearson with Spearman correlation
• Thinking Pearson measures differences in means
• Assuming Pearson works for non-linear relationships

2. Which of the following Python code snippets correctly calculates the Spearman correlation between two pandas Series x and y?

easy

A. x.corr(y, method='spearman')

B. x.corr(y, method='pearson')

C. spearmanr(x, y)

D. corr(x, y, method='spearman')

Solution

1. Step 1: Recall pandas correlation syntax

   In pandas, Series.corr() accepts a method argument with values like 'pearson' or 'spearman'.

2. Step 2: Identify correct method call

   x.corr(y, method='spearman') uses x.corr(y, method='spearman'), which is the correct pandas syntax.

3. Final Answer:

   x.corr(y, method='spearman') -> Option A

4. Quick Check:

   pandas Series.corr with method='spearman' [OK]

Hint: Use Series.corr with method='spearman' for Spearman correlation [OK]

Common Mistakes:

• Using 'pearson' instead of 'spearman' for Spearman correlation
• Calling a non-existent function like corr() standalone
• Confusing scipy's spearmanr with pandas syntax

3. Given the following code, what is the output of print(round(corr, 2))?

import pandas as pd
x = pd.Series([1, 2, 3, 4, 5])
y = pd.Series([5, 6, 7, 8, 8])
corr = x.corr(y, method='pearson')

medium

A. 0.75

B. 0.90

C. 0.85

D. 0.97

Solution

1. Step 1: Calculate Pearson correlation manually

   Calculate covariance and standard deviations of x and y, then compute correlation. Here, x and y increase mostly together, so correlation is high.

2. Step 2: Use pandas to confirm

   Running the code gives approximately 0.9746, which rounds to 0.97.

3. Final Answer:

   0.97 -> Option D

4. Quick Check:

   pandas Pearson correlation ≈ 0.97 [OK]

Hint: Use pandas corr() to get exact correlation quickly [OK]

Common Mistakes:

• Confusing Pearson with Spearman correlation values
• Rounding incorrectly or too early
• Assuming correlation must be exactly 1 for similar sequences

4. Identify the error in this code snippet that attempts to calculate Spearman correlation:

from scipy.stats import spearmanr
x = [1, 2, 3, 4]
y = [4, 3, 2, 1]
result = spearmanr(x, y)
print(result.correlation)

medium

A. The function spearmanr requires numpy arrays, not lists

B. The attribute 'correlation' does not exist in the result object

C. The print statement should be inside a function

D. The import statement is incorrect

Solution

1. Step 1: Check spearmanr return type

   spearmanr returns a namedtuple SpearmanRResult with fields statistic (Spearman correlation coefficient) and pvalue.

2. Step 2: Confirm attribute access

   The code accesses result.correlation, but no such attribute exists. Use result.statistic instead.

3. Final Answer:

   The attribute 'correlation' does not exist in the result object -> Option B

4. Quick Check:

   spearmanr uses result.statistic [OK]

Hint: Use result.statistic for spearmanr correlation [OK]

Common Mistakes:

• Assuming lists cannot be used with spearmanr
• Misunderstanding the return type of spearmanr
• Incorrect import or syntax errors

5. You have two variables: age (numeric) and education_level (ordinal ranks: 1=High School, 2=Bachelor, 3=Master, 4=PhD). Which correlation method is most appropriate to measure their association and why?

hard

A. Spearman, because education_level is ordinal and relationship may be monotonic

B. Pearson, because it handles ordinal data better

C. Neither, because correlation cannot be calculated between these variables

D. Pearson, because both are numeric variables

Solution

1. Step 1: Identify variable types

   Age is numeric continuous; education_level is ordinal categorical (ranked).

2. Step 2: Choose correlation method

   Spearman correlation is best for ordinal data and monotonic relationships, unlike Pearson which assumes linear and interval data.

3. Final Answer:

   Spearman, because education_level is ordinal and relationship may be monotonic -> Option A

4. Quick Check:

   Ordinal data + monotonic = Spearman [OK]

Hint: Use Spearman for ordinal or ranked data [OK]

Common Mistakes:

• Using Pearson for ordinal data incorrectly
• Thinking correlation can't be done with ordinal variables
• Confusing monotonic with linear relationships