Correlation analysis in ML Python - ML Experiment: Train & Evaluate

Experiment - Correlation analysis

Problem:You want to understand how different features in your dataset relate to each other and to the target variable. Currently, you have a dataset with multiple features but no clear insight into which features are most important or how they interact.

Current Metrics:No correlation analysis done yet, so no metrics available.

Issue:Without correlation analysis, you risk using irrelevant or redundant features that can confuse your model and reduce accuracy.

Your Task

Perform correlation analysis on the dataset to identify strong positive or negative relationships between features and the target variable. Use this insight to select the most relevant features for a simple model.

You must use Pearson correlation coefficient.

You cannot use automated feature selection libraries.

You should visualize the correlation matrix using a heatmap.

Hint 1

Hint 2

Hint 3

Solution

ML Python

import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt

# Sample dataset creation
# Imagine this is your dataset with features and a target
data = {
    'Feature1': [10, 20, 30, 40, 50],
    'Feature2': [15, 25, 35, 45, 55],
    'Feature3': [5, 3, 6, 2, 1],
    'Target': [100, 200, 300, 400, 500]
}
df = pd.DataFrame(data)

# Calculate correlation matrix
corr_matrix = df.corr(method='pearson')

# Print correlation matrix
print('Correlation matrix:')
print(corr_matrix)

# Visualize correlation matrix with heatmap
sns.heatmap(corr_matrix, annot=True, cmap='coolwarm')
plt.title('Feature Correlation Heatmap')
plt.show()

Added code to calculate Pearson correlation matrix using pandas.

Added heatmap visualization using seaborn for easy interpretation.

Selected features based on correlation strength with the target variable.

Results Interpretation

Before: No understanding of feature relationships, risk of poor feature choice.

After: Clear correlation values and heatmap show which features relate strongly to the target. This helps pick better features for modeling.

Correlation analysis helps identify which features are most related to the target. This simple step can improve model accuracy by focusing on important features and ignoring irrelevant ones.

Bonus Experiment

Try using Spearman correlation instead of Pearson to see how it handles non-linear relationships.

💡 Hint

Use pandas .corr(method='spearman') and compare the heatmap results with Pearson.

Practice

1. What does a correlation value of 0 indicate between two variables?

easy

A. There is no linear relationship between the variables

B. The variables move exactly together in the same direction

C. One variable increases while the other decreases perfectly

D. The variables have a perfect nonlinear relationship

Solution

1. Step 1: Understand correlation meaning

   Correlation measures linear relationship strength between two variables.

2. Step 2: Interpret zero correlation

   A correlation of 0 means no linear relationship exists between the variables.

3. Final Answer:

   There is no linear relationship between the variables -> Option A

4. Quick Check:

   Correlation = 0 means no linear link [OK]

Hint: Zero correlation means no linear link between variables [OK]

Common Mistakes:

• Thinking zero means variables are unrelated in any way
• Confusing zero correlation with negative correlation
• Assuming zero means perfect nonlinear relationship

2. Which of the following is the correct Python syntax to calculate Pearson correlation between two lists x and y using NumPy?

easy

A. np.correlation(x, y)

B. np.corrcoef(x, y)[0, 1]

C. np.pearson(x, y)

D. np.corr(x, y)

Solution

1. Step 1: Recall NumPy correlation function

   NumPy uses np.corrcoef() to compute correlation matrix.

2. Step 2: Extract Pearson correlation value

   The correlation between x and y is at position [0, 1] in the matrix.

3. Final Answer:

   np.corrcoef(x, y)[0, 1] -> Option B

4. Quick Check:

   Use np.corrcoef and index [0,1] for Pearson [OK]

Hint: Use np.corrcoef(x, y)[0,1] to get Pearson correlation [OK]

Common Mistakes:

• Using non-existent functions like np.correlation or np.pearson
• Forgetting to index the correlation matrix
• Confusing correlation with covariance functions

3. Given the code below, what is the output?

import numpy as np
x = [1, 2, 3, 4, 5]
y = [2, 4, 6, 8, 10]
correlation = np.corrcoef(x, y)[0, 1]
print("{:.2f}".format(round(correlation, 2)))

medium

A. 1.00

B. -1.00

C. 0.00

D. 0.50

Solution

1. Step 1: Analyze the data relationship

   List y is exactly twice x, so they have a perfect positive linear relationship.

2. Step 2: Calculate correlation

   Perfect positive linear relationship means correlation = 1.0.

3. Final Answer:

   1.00 -> Option A

4. Quick Check:

   Perfect positive linear relation = 1.00 [OK]

Hint: Perfect multiples have correlation 1.00 [OK]

Common Mistakes:

• Thinking correlation is 0 because values differ
• Confusing correlation with covariance magnitude
• Rounding errors causing wrong output

4. Identify the error in the following code snippet for calculating correlation:

import numpy as np
x = [1, 2, 3]
y = [4, 5]
correlation = np.corrcoef(x, y)[0, 1]
print(correlation)

medium

A. Missing import statement for correlation function

B. np.corrcoef requires pandas Series, not lists

C. Lists x and y have different lengths causing an error

D. Indexing [0, 1] is incorrect for correlation matrix

Solution

1. Step 1: Check input data lengths

   Lists x and y have lengths 3 and 2, which must be equal for correlation.

2. Step 2: Understand np.corrcoef requirements

   np.corrcoef requires inputs of equal length arrays to compute correlation matrix.

3. Final Answer:

   Lists x and y have different lengths causing an error -> Option C

4. Quick Check:

   Input arrays must be same length [OK]

Hint: Check input lengths match before correlation [OK]

Common Mistakes:

• Assuming np.corrcoef works with unequal length lists
• Thinking pandas Series are required for np.corrcoef
• Misindexing the correlation matrix

5. You have two features: hours_studied and exam_score. You calculate their correlation and get -0.85. What does this imply and what should you consider next?

hard

A. Perfect negative nonlinear relationship; use polynomial regression

B. Strong positive linear relationship; increase study hours to improve score

C. No relationship; ignore this feature in modeling

D. Strong negative linear relationship; consider if data or labels are reversed

Solution

1. Step 1: Interpret correlation sign and magnitude

   A correlation of -0.85 means a strong negative linear relationship between the two features.

2. Step 2: Consider real-world meaning

   Since more study usually means higher scores, a strong negative correlation suggests possible data issues or reversed labels.

3. Final Answer:

   Strong negative linear relationship; consider if data or labels are reversed -> Option D

4. Quick Check:

   Negative strong correlation may indicate reversed data [OK]

Hint: Strong negative correlation may mean reversed data [OK]

Common Mistakes:

• Assuming negative correlation means no relationship
• Ignoring domain knowledge about expected relationships
• Confusing negative linear with nonlinear relationships