Correlation with corr() in Data Analysis Python - Mini Project: Build & Apply

Correlation with corr()

📖 Scenario: You are a data analyst working with a small dataset of students' scores in two subjects: Math and English. You want to find out if there is a relationship between their scores in these subjects.

🎯 Goal: Build a simple Python program that creates a dataset of students' scores, sets up a DataFrame, calculates the correlation between Math and English scores using corr(), and prints the result.

📋 What You'll Learn

Create a dictionary called scores with keys 'Math' and 'English' and the exact values: [88, 92, 79, 93, 85] and [84, 90, 78, 88, 82] respectively

Import pandas as pd

Create a DataFrame called df from the scores dictionary

Calculate the correlation between the 'Math' and 'English' columns using df['Math'].corr(df['English']) and store it in a variable called correlation

Print the correlation variable

💡 Why This Matters

🌍 Real World

Correlation helps to find relationships between variables, like how scores in one subject relate to another. This is useful in education, business, and science.

💼 Career

Data analysts and scientists use correlation to understand data patterns and make decisions based on relationships between variables.

1

Create the scores dictionary

Create a dictionary called scores with keys 'Math' and 'English'. Set the value for 'Math' to [88, 92, 79, 93, 85] and for 'English' to [84, 90, 78, 88, 82].

Data Analysis Python

# Create the scores dictionary with exact values
# Your code here

Hint

Use curly braces {} to create a dictionary. The keys are 'Math' and 'English'. The values are lists of numbers.

2

Import pandas and create DataFrame

Import the pandas library as pd. Then create a DataFrame called df from the scores dictionary using pd.DataFrame(scores).

Data Analysis Python

scores = {'Math': [88, 92, 79, 93, 85], 'English': [84, 90, 78, 88, 82]}
# Import pandas as pd and create DataFrame df from scores
# Your code here

Hint

Use import pandas as pd to import pandas. Then use pd.DataFrame() to create the DataFrame.

3

Calculate correlation between Math and English

Calculate the correlation between the 'Math' and 'English' columns in df using df['Math'].corr(df['English']). Store the result in a variable called correlation.

Data Analysis Python

scores = {'Math': [88, 92, 79, 93, 85], 'English': [84, 90, 78, 88, 82]}
import pandas as pd
df = pd.DataFrame(scores)
# Calculate correlation between 'Math' and 'English' columns
# Your code here

Hint

Use the corr() method on the 'Math' column and pass the 'English' column as argument.

4

Print the correlation result

Print the variable correlation to display the correlation value between Math and English scores.

Data Analysis Python

scores = {'Math': [88, 92, 79, 93, 85], 'English': [84, 90, 78, 88, 82]}
import pandas as pd
df = pd.DataFrame(scores)
correlation = df['Math'].corr(df['English'])
# Print the correlation variable
# Your code here

Hint

Use print(correlation) to show the correlation value.

Practice

1. What does the corr() function in pandas do?

easy

A. Counts the number of missing values in a DataFrame

B. Calculates the correlation between numeric columns in a DataFrame

C. Sorts the DataFrame by a specific column

D. Filters rows based on a condition

Solution

1. Step 1: Understand the purpose of corr()

   The corr() function calculates how numeric columns relate to each other in a DataFrame.

2. Step 2: Compare with other options

   Counting missing values, sorting, or filtering are different functions like isna(), sort_values(), and loc.

3. Final Answer:

   Calculates the correlation between numeric columns in a DataFrame -> Option B

4. Quick Check:

   corr() finds relationships [OK]

Hint: Remember: corr() measures how numbers move together [OK]

Common Mistakes:

• Confusing corr() with counting or sorting functions
• Thinking corr() works on non-numeric data
• Assuming corr() changes the DataFrame structure

2. Which of the following is the correct way to calculate correlation between columns in a pandas DataFrame named df?

easy

A. df.corr()

B. df.correlate()

C. df.corrrelation()

D. df.correlation()

Solution

1. Step 1: Recall the exact method name

   The correct pandas method to calculate correlation is corr().

2. Step 2: Check each option

   Options A, B, and C are misspelled or incorrect method names.

3. Final Answer:

   df.corr() -> Option A

4. Quick Check:

   Use df.corr() for correlation [OK]

Hint: Remember exact method name: corr() not correlation [OK]

Common Mistakes:

• Misspelling the method name
• Using non-existent pandas methods
• Confusing corr() with other functions

3. Given the following DataFrame df:

   A  B  C
0  1  2  5
1  2  4  7
2  3  6  9
3  4  8  11

What is the output of df.corr()?

medium

A. {'A': {'A': 1.0, 'B': 1.0, 'C': 1.0}, 'B': {'A': 1.0, 'B': 1.0, 'C': 1.0}, 'C': {'A': 1.0, 'B': 1.0, 'C': 1.0}}

B. {'A': {'A': 1.0, 'B': 0.5, 'C': 0.5}, 'B': {'A': 0.5, 'B': 1.0, 'C': 0.5}, 'C': {'A': 0.5, 'B': 0.5, 'C': 1.0}}

C. {'A': {'A': 1.0, 'B': -1.0, 'C': 0.0}, 'B': {'A': -1.0, 'B': 1.0, 'C': 0.0}, 'C': {'A': 0.0, 'B': 0.0, 'C': 1.0}}

D. Error: corr() cannot be applied to this DataFrame

Solution

1. Step 1: Understand the data pattern

   Columns A, B, and C increase perfectly linearly: B = 2*A, C = 2*A + 3.

2. Step 2: Calculate correlation values

   Since all columns are perfectly linearly related, correlation between any two is 1.0, and correlation with itself is 1.0.

3. Final Answer:

   {'A': {'A': 1.0, 'B': 1.0, 'C': 1.0}, 'B': {'A': 1.0, 'B': 1.0, 'C': 1.0}, 'C': {'A': 1.0, 'B': 1.0, 'C': 1.0}} -> Option A

4. Quick Check:

   Perfect linear relation means correlation = 1 [OK]

Hint: Perfect linear columns have correlation 1 [OK]

Common Mistakes:

• Assuming correlation less than 1 for perfect linear data
• Confusing correlation with covariance
• Thinking corr() raises error on numeric data

4. You run the code below but get an error:

import pandas as pd
df = pd.DataFrame({'X': [1, 2, 3], 'Y': ['a', 'b', 'c']})
print(df.corr(numeric_only=False))

What is the cause of the error?

medium

A. The corr() method is misspelled

B. The DataFrame is empty

C. The DataFrame has non-numeric columns which corr() cannot process

D. The DataFrame columns have missing values

Solution

1. Step 1: Analyze DataFrame columns

   Column 'Y' contains strings, which are non-numeric.

2. Step 2: Understand corr() requirements

   corr() only works on numeric columns; non-numeric columns cause it to ignore or error depending on pandas version.

3. Final Answer:

   The DataFrame has non-numeric columns which corr() cannot process -> Option C

4. Quick Check:

   corr() needs numeric data [OK]

Hint: Check for non-numeric columns before using corr() [OK]

Common Mistakes:

• Assuming corr() works on strings
• Thinking misspelling caused error
• Ignoring data types in DataFrame

5. You have a DataFrame df with columns height, weight, and age. You want to find which two variables have the strongest positive correlation. Which code snippet correctly finds this?

hard

A. corr_matrix = df.corr()
max_corr = corr_matrix.where(corr_matrix < 1).max().max()

B. corr_matrix = df.corr()
max_corr = corr_matrix.where(corr_matrix != 1).min().min()

C. corr_matrix = df.corr()
max_corr = corr_matrix.max().max()

D. corr_matrix = df.corr()
max_corr = corr_matrix.where(corr_matrix != 1).max().max()

Solution

1. Step 1: Calculate correlation matrix

   Use df.corr() to get correlations between all numeric columns.

2. Step 2: Exclude self-correlation (which is 1)

   Use where(corr_matrix != 1) to ignore diagonal 1s, then find the maximum correlation value.

3. Step 3: Find the strongest positive correlation

   Use max().max() to get the highest correlation excluding 1.

4. Final Answer:

   corr_matrix = df.corr() max_corr = corr_matrix.where(corr_matrix != 1).max().max() -> Option D

5. Quick Check:

   Exclude 1, then find max correlation [OK]

Hint: Exclude 1s (self-correlation) before max() [OK]

Common Mistakes:

• Including diagonal 1s in max correlation
• Using min() instead of max()
• Not excluding self-correlation values