Correlation with corr() in Data Analysis Python - Step-by-Step Execution

Concept Flow - Correlation with corr()

Start with DataFrame

↓

Call corr() method

↓

Calculate pairwise correlations

↓

Return correlation matrix

↓

Use or visualize results

This flow shows how calling corr() on a DataFrame calculates pairwise correlations and returns a matrix.

Execution Sample

Data Analysis Python

import pandas as pd

data = {'A': [1, 2, 3], 'B': [4, 5, 6], 'C': [7, 8, 9]}
df = pd.DataFrame(data)
correlation_matrix = df.corr()
print(correlation_matrix)

This code creates a DataFrame and calculates the correlation matrix between its columns.

Execution Table

Step	Action	Calculation	Result
1	Create DataFrame	Columns A, B, C with values	df with 3 rows and 3 columns
2	Call df.corr()	Calculate pairwise Pearson correlations	Compute correlation for (A,B), (A,C), (B,C) and diagonals
3	Correlation A vs A	Correlation of A with itself	1.0
4	Correlation A vs B	Calculate correlation coefficient	1.0
5	Correlation A vs C	Calculate correlation coefficient	1.0
6	Correlation B vs B	Correlation of B with itself	1.0
7	Correlation B vs C	Calculate correlation coefficient	1.0
8	Correlation C vs C	Correlation of C with itself	1.0
9	Return matrix	Assemble all correlations into matrix	3x3 matrix with all 1.0 values

💡 All pairwise correlations computed; matrix returned.

Variable Tracker

Variable	Start	After corr() call	Final
df	{'A':[1,2,3],'B':[4,5,6],'C':[7,8,9]}	Same DataFrame	Same DataFrame
correlation_matrix	None	3x3 matrix with 1.0 in all cells	3x3 matrix with 1.0 in all cells

Key Moments - 2 Insights

Why are all correlation values 1.0 in the matrix?

Because columns A, B, and C increase perfectly together, their Pearson correlation is 1.0 as shown in steps 4, 5, and 7 of the execution_table.

Does corr() change the original DataFrame?

No, corr() only calculates and returns a new correlation matrix without modifying the original DataFrame, as seen in variable_tracker where df remains unchanged.

Visual Quiz - 3 Questions

Look at the execution_table at step 4, what is the correlation between columns A and B?

A 0.5

B 1.0

C -1.0

D 0.0

💡 Hint

Refer to execution_table row with Step 4 showing correlation A vs B.

At which step does the code calculate the correlation of column B with itself?

A Step 6

B Step 7

C Step 3

D Step 9

💡 Hint

Look at execution_table row labeled 'Correlation B vs B'.

If column C had random values instead of increasing values, how would the correlation_matrix change?

A All values would still be 1.0

B Diagonal values would be 0

C Some off-diagonal values would be less than 1.0

D Matrix would be empty

💡 Hint

Correlation depends on how columns relate; see variable_tracker and execution_table for perfect correlation case.

Concept Snapshot

DataFrame.corr() computes pairwise correlation coefficients.
Returns a matrix showing correlation between each pair of columns.
Values range from -1 (inverse) to 1 (perfect positive).
Diagonal values are always 1 (column with itself).
Useful to find relationships between numeric columns.

Full Transcript

We start with a DataFrame containing numeric columns A, B, and C. Calling the corr() method calculates the Pearson correlation coefficient between each pair of columns. The method returns a new DataFrame showing these correlations. In this example, since all columns increase together perfectly, all correlation values are 1.0. The original DataFrame remains unchanged. This process helps us understand how strongly columns relate to each other.

Practice

1. What does the corr() function in pandas do?

easy

A. Counts the number of missing values in a DataFrame

B. Calculates the correlation between numeric columns in a DataFrame

C. Sorts the DataFrame by a specific column

D. Filters rows based on a condition

Solution

1. Step 1: Understand the purpose of corr()

   The corr() function calculates how numeric columns relate to each other in a DataFrame.

2. Step 2: Compare with other options

   Counting missing values, sorting, or filtering are different functions like isna(), sort_values(), and loc.

3. Final Answer:

   Calculates the correlation between numeric columns in a DataFrame -> Option B

4. Quick Check:

   corr() finds relationships [OK]

Hint: Remember: corr() measures how numbers move together [OK]

Common Mistakes:

• Confusing corr() with counting or sorting functions
• Thinking corr() works on non-numeric data
• Assuming corr() changes the DataFrame structure

2. Which of the following is the correct way to calculate correlation between columns in a pandas DataFrame named df?

easy

A. df.corr()

B. df.correlate()

C. df.corrrelation()

D. df.correlation()

Solution

1. Step 1: Recall the exact method name

   The correct pandas method to calculate correlation is corr().

2. Step 2: Check each option

   Options A, B, and C are misspelled or incorrect method names.

3. Final Answer:

   df.corr() -> Option A

4. Quick Check:

   Use df.corr() for correlation [OK]

Hint: Remember exact method name: corr() not correlation [OK]

Common Mistakes:

• Misspelling the method name
• Using non-existent pandas methods
• Confusing corr() with other functions

3. Given the following DataFrame df:

   A  B  C
0  1  2  5
1  2  4  7
2  3  6  9
3  4  8  11

What is the output of df.corr()?

medium

A. {'A': {'A': 1.0, 'B': 1.0, 'C': 1.0}, 'B': {'A': 1.0, 'B': 1.0, 'C': 1.0}, 'C': {'A': 1.0, 'B': 1.0, 'C': 1.0}}

B. {'A': {'A': 1.0, 'B': 0.5, 'C': 0.5}, 'B': {'A': 0.5, 'B': 1.0, 'C': 0.5}, 'C': {'A': 0.5, 'B': 0.5, 'C': 1.0}}

C. {'A': {'A': 1.0, 'B': -1.0, 'C': 0.0}, 'B': {'A': -1.0, 'B': 1.0, 'C': 0.0}, 'C': {'A': 0.0, 'B': 0.0, 'C': 1.0}}

D. Error: corr() cannot be applied to this DataFrame

Solution

1. Step 1: Understand the data pattern

   Columns A, B, and C increase perfectly linearly: B = 2*A, C = 2*A + 3.

2. Step 2: Calculate correlation values

   Since all columns are perfectly linearly related, correlation between any two is 1.0, and correlation with itself is 1.0.

3. Final Answer:

   {'A': {'A': 1.0, 'B': 1.0, 'C': 1.0}, 'B': {'A': 1.0, 'B': 1.0, 'C': 1.0}, 'C': {'A': 1.0, 'B': 1.0, 'C': 1.0}} -> Option A

4. Quick Check:

   Perfect linear relation means correlation = 1 [OK]

Hint: Perfect linear columns have correlation 1 [OK]

Common Mistakes:

• Assuming correlation less than 1 for perfect linear data
• Confusing correlation with covariance
• Thinking corr() raises error on numeric data

4. You run the code below but get an error:

import pandas as pd
df = pd.DataFrame({'X': [1, 2, 3], 'Y': ['a', 'b', 'c']})
print(df.corr(numeric_only=False))

What is the cause of the error?

medium

A. The corr() method is misspelled

B. The DataFrame is empty

C. The DataFrame has non-numeric columns which corr() cannot process

D. The DataFrame columns have missing values

Solution

1. Step 1: Analyze DataFrame columns

   Column 'Y' contains strings, which are non-numeric.

2. Step 2: Understand corr() requirements

   corr() only works on numeric columns; non-numeric columns cause it to ignore or error depending on pandas version.

3. Final Answer:

   The DataFrame has non-numeric columns which corr() cannot process -> Option C

4. Quick Check:

   corr() needs numeric data [OK]

Hint: Check for non-numeric columns before using corr() [OK]

Common Mistakes:

• Assuming corr() works on strings
• Thinking misspelling caused error
• Ignoring data types in DataFrame

5. You have a DataFrame df with columns height, weight, and age. You want to find which two variables have the strongest positive correlation. Which code snippet correctly finds this?

hard

A. corr_matrix = df.corr()
max_corr = corr_matrix.where(corr_matrix < 1).max().max()

B. corr_matrix = df.corr()
max_corr = corr_matrix.where(corr_matrix != 1).min().min()

C. corr_matrix = df.corr()
max_corr = corr_matrix.max().max()

D. corr_matrix = df.corr()
max_corr = corr_matrix.where(corr_matrix != 1).max().max()

Solution

1. Step 1: Calculate correlation matrix

   Use df.corr() to get correlations between all numeric columns.

2. Step 2: Exclude self-correlation (which is 1)

   Use where(corr_matrix != 1) to ignore diagonal 1s, then find the maximum correlation value.

3. Step 3: Find the strongest positive correlation

   Use max().max() to get the highest correlation excluding 1.

4. Final Answer:

   corr_matrix = df.corr() max_corr = corr_matrix.where(corr_matrix != 1).max().max() -> Option D

5. Quick Check:

   Exclude 1, then find max correlation [OK]

Hint: Exclude 1s (self-correlation) before max() [OK]

Common Mistakes:

• Including diagonal 1s in max correlation
• Using min() instead of max()
• Not excluding self-correlation values