Correlation with corr() in Data Analysis Python

Introduction

Correlation helps us see how two things change together. It tells if one thing goes up when the other goes up or down.

To check if study time and test scores are related.

To find if temperature and ice cream sales move together.

To see if height and weight have a connection.

To understand if advertising budget affects sales.

To explore relationships between different features in a dataset before modeling.

Syntax

Data Analysis Python

DataFrame.corr(method='pearson')

This method calculates correlation between numeric columns in a DataFrame.

The default method is 'pearson', which measures linear correlation.

Examples

Calculate Pearson correlation between all numeric columns in the DataFrame df.

Data Analysis Python

df.corr()

Explicitly specify Pearson correlation (default).

Data Analysis Python

df.corr(method='pearson')

Calculate Kendall Tau correlation, useful for small datasets or non-linear relationships.

Data Analysis Python

df.corr(method='kendall')

Calculate Spearman rank correlation, which measures monotonic relationships.

Data Analysis Python

df.corr(method='spearman')

Sample Program

This code creates a small dataset with study hours, test scores, and sleep hours. Then it calculates the correlation matrix to see how these columns relate.

Data Analysis Python

import pandas as pd

# Create a simple dataset
data = {
    'study_hours': [2, 3, 5, 8, 10],
    'test_score': [50, 55, 65, 80, 90],
    'sleep_hours': [7, 6, 5, 6, 7]
}
df = pd.DataFrame(data)

# Calculate correlation matrix
corr_matrix = df.corr()
print(corr_matrix)

Important Notes

Correlation values range from -1 to 1.

A value close to 1 means strong positive relation, close to -1 means strong negative relation, and around 0 means no relation.

Only numeric columns are used in correlation calculation.

Summary

Use corr() to find relationships between numeric columns.

Correlation helps understand how variables move together.

Values near 1 or -1 show strong relationships; near 0 means weak or no relationship.

Practice

1. What does the corr() function in pandas do?

easy

A. Counts the number of missing values in a DataFrame

B. Calculates the correlation between numeric columns in a DataFrame

C. Sorts the DataFrame by a specific column

D. Filters rows based on a condition

Solution

1. Step 1: Understand the purpose of corr()

   The corr() function calculates how numeric columns relate to each other in a DataFrame.

2. Step 2: Compare with other options

   Counting missing values, sorting, or filtering are different functions like isna(), sort_values(), and loc.

3. Final Answer:

   Calculates the correlation between numeric columns in a DataFrame -> Option B

4. Quick Check:

   corr() finds relationships [OK]

Hint: Remember: corr() measures how numbers move together [OK]

Common Mistakes:

• Confusing corr() with counting or sorting functions
• Thinking corr() works on non-numeric data
• Assuming corr() changes the DataFrame structure

2. Which of the following is the correct way to calculate correlation between columns in a pandas DataFrame named df?

easy

A. df.corr()

B. df.correlate()

C. df.corrrelation()

D. df.correlation()

Solution

1. Step 1: Recall the exact method name

   The correct pandas method to calculate correlation is corr().

2. Step 2: Check each option

   Options A, B, and C are misspelled or incorrect method names.

3. Final Answer:

   df.corr() -> Option A

4. Quick Check:

   Use df.corr() for correlation [OK]

Hint: Remember exact method name: corr() not correlation [OK]

Common Mistakes:

• Misspelling the method name
• Using non-existent pandas methods
• Confusing corr() with other functions

3. Given the following DataFrame df:

   A  B  C
0  1  2  5
1  2  4  7
2  3  6  9
3  4  8  11

What is the output of df.corr()?

medium

A. {'A': {'A': 1.0, 'B': 1.0, 'C': 1.0}, 'B': {'A': 1.0, 'B': 1.0, 'C': 1.0}, 'C': {'A': 1.0, 'B': 1.0, 'C': 1.0}}

B. {'A': {'A': 1.0, 'B': 0.5, 'C': 0.5}, 'B': {'A': 0.5, 'B': 1.0, 'C': 0.5}, 'C': {'A': 0.5, 'B': 0.5, 'C': 1.0}}

C. {'A': {'A': 1.0, 'B': -1.0, 'C': 0.0}, 'B': {'A': -1.0, 'B': 1.0, 'C': 0.0}, 'C': {'A': 0.0, 'B': 0.0, 'C': 1.0}}

D. Error: corr() cannot be applied to this DataFrame

Solution

1. Step 1: Understand the data pattern

   Columns A, B, and C increase perfectly linearly: B = 2*A, C = 2*A + 3.

2. Step 2: Calculate correlation values

   Since all columns are perfectly linearly related, correlation between any two is 1.0, and correlation with itself is 1.0.

3. Final Answer:

   {'A': {'A': 1.0, 'B': 1.0, 'C': 1.0}, 'B': {'A': 1.0, 'B': 1.0, 'C': 1.0}, 'C': {'A': 1.0, 'B': 1.0, 'C': 1.0}} -> Option A

4. Quick Check:

   Perfect linear relation means correlation = 1 [OK]

Hint: Perfect linear columns have correlation 1 [OK]

Common Mistakes:

• Assuming correlation less than 1 for perfect linear data
• Confusing correlation with covariance
• Thinking corr() raises error on numeric data

4. You run the code below but get an error:

import pandas as pd
df = pd.DataFrame({'X': [1, 2, 3], 'Y': ['a', 'b', 'c']})
print(df.corr(numeric_only=False))

What is the cause of the error?

medium

A. The corr() method is misspelled

B. The DataFrame is empty

C. The DataFrame has non-numeric columns which corr() cannot process

D. The DataFrame columns have missing values

Solution

1. Step 1: Analyze DataFrame columns

   Column 'Y' contains strings, which are non-numeric.

2. Step 2: Understand corr() requirements

   corr() only works on numeric columns; non-numeric columns cause it to ignore or error depending on pandas version.

3. Final Answer:

   The DataFrame has non-numeric columns which corr() cannot process -> Option C

4. Quick Check:

   corr() needs numeric data [OK]

Hint: Check for non-numeric columns before using corr() [OK]

Common Mistakes:

• Assuming corr() works on strings
• Thinking misspelling caused error
• Ignoring data types in DataFrame

5. You have a DataFrame df with columns height, weight, and age. You want to find which two variables have the strongest positive correlation. Which code snippet correctly finds this?

hard

A. corr_matrix = df.corr()
max_corr = corr_matrix.where(corr_matrix < 1).max().max()

B. corr_matrix = df.corr()
max_corr = corr_matrix.where(corr_matrix != 1).min().min()

C. corr_matrix = df.corr()
max_corr = corr_matrix.max().max()

D. corr_matrix = df.corr()
max_corr = corr_matrix.where(corr_matrix != 1).max().max()

Solution

1. Step 1: Calculate correlation matrix

   Use df.corr() to get correlations between all numeric columns.

2. Step 2: Exclude self-correlation (which is 1)

   Use where(corr_matrix != 1) to ignore diagonal 1s, then find the maximum correlation value.

3. Step 3: Find the strongest positive correlation

   Use max().max() to get the highest correlation excluding 1.

4. Final Answer:

   corr_matrix = df.corr() max_corr = corr_matrix.where(corr_matrix != 1).max().max() -> Option D

5. Quick Check:

   Exclude 1, then find max correlation [OK]

Hint: Exclude 1s (self-correlation) before max() [OK]

Common Mistakes:

• Including diagonal 1s in max correlation
• Using min() instead of max()
• Not excluding self-correlation values