Pearson correlation in SciPy - Deep Dive

Overview - Pearson correlation

What is it?

Pearson correlation is a way to measure how two sets of numbers move together. It tells us if when one number goes up, the other tends to go up or down, and how strong that relationship is. The result is a number between -1 and 1. A value close to 1 means they move up together, close to -1 means they move in opposite directions, and around 0 means no clear relationship.

Why it matters

Without Pearson correlation, it would be hard to understand relationships between data points in many fields like health, finance, or social sciences. It helps us find patterns and connections that guide decisions, like knowing if studying more relates to better grades. Without it, we might guess blindly and miss important insights.

Where it fits

Before learning Pearson correlation, you should understand basic statistics like mean and standard deviation. After this, you can explore more complex relationships with other correlation types or regression analysis to predict outcomes.

Mental Model

Core Idea

Pearson correlation measures how two variables linearly move together, quantifying the strength and direction of their relationship.

Think of it like...

Imagine two dancers moving on a stage. If they move perfectly in sync, their dance is like a correlation of 1. If one moves left while the other moves right exactly, that's like -1. If their moves don’t match at all, it’s like zero correlation.

Variables X and Y relationship:

  +1 ──────────────── Perfect positive linear relationship
   |
   |      *
   |     * *
   |    *   *
   |   *     *
  0|------------------ No linear relationship
   | *       *
   |  *     *
   |   *   *
   |    * *
  -1 ──────────────── Perfect negative linear relationship

Build-Up - 7 Steps

1

Foundation

Understanding variables and data pairs

Concept: Learn what variables are and how data points come in pairs for correlation.

Variables are things we measure, like height or test scores. To find correlation, we need pairs of numbers, one from each variable, measured from the same subject or time. For example, height and weight of the same person form a pair.

Result

You can organize data as pairs, ready to compare how they move together.

Knowing data must be paired correctly is key to measuring any relationship between variables.

2

Foundation

Calculating mean and standard deviation

Concept: Learn how to find the average and spread of data, which are building blocks for correlation.

Mean is the average value of a variable. Standard deviation shows how spread out the data is around the mean. Both are needed to standardize data before comparing two variables.

Result

You can summarize each variable’s center and spread, preparing for comparison.

Understanding mean and spread helps us see how data points relate to their average, which is essential for correlation.

3

Intermediate

Formula for Pearson correlation coefficient

🤔Before reading on: do you think Pearson correlation measures any relationship or only linear ones? Commit to your answer.

Concept: Pearson correlation uses a formula that compares how each pair of values deviates from their means, scaled by their spreads.

The formula is: r = sum((x_i - mean_x) * (y_i - mean_y)) / ((n-1) * std_x * std_y) where x_i and y_i are paired values, mean_x and mean_y are averages, std_x and std_y are standard deviations, and n is number of pairs.

Result

You get a number between -1 and 1 showing the strength and direction of a linear relationship.

Knowing the formula reveals that correlation is about how paired deviations align, not just raw values.

4

Intermediate

Using scipy to compute Pearson correlation

🤔Before reading on: do you think scipy returns just the correlation number or more information? Commit to your answer.

Concept: The scipy library provides a simple function to calculate Pearson correlation and its significance.

Example code: import scipy.stats as stats x = [1, 2, 3, 4, 5] y = [2, 4, 6, 8, 10] corr, p_value = stats.pearsonr(x, y) print(f"Correlation: {corr}") print(f"P-value: {p_value}")

Result

Output: Correlation: 1.0 P-value: 0.0 This shows perfect positive correlation and a very significant result.

Using scipy simplifies calculation and adds statistical significance, helping decide if correlation is meaningful.

5

Intermediate

Interpreting correlation and p-value

🤔Before reading on: do you think a high correlation always means one variable causes the other? Commit to your answer.

Concept: Correlation shows strength and direction but not cause. P-value tells if the correlation is likely due to chance.

A correlation near 1 or -1 means strong linear relationship. P-value below 0.05 usually means the result is statistically significant. But correlation does not prove one variable causes the other.

Result

You can judge if a relationship is strong and reliable, but must be careful about cause and effect.

Understanding the limits of correlation prevents wrong conclusions about cause.

6

Advanced

Limitations and assumptions of Pearson correlation

🤔Before reading on: do you think Pearson correlation works well with any data shape? Commit to your answer.

Concept: Pearson correlation assumes a linear relationship and data without extreme outliers or non-normal distribution.

If data is curved, has outliers, or is not normally distributed, Pearson correlation can be misleading. Alternatives like Spearman correlation or data transformation may be better.

Result

You learn when Pearson correlation is appropriate and when to choose other methods.

Knowing assumptions helps avoid misusing Pearson correlation and drawing wrong insights.

7

Expert

Pearson correlation in multivariate and big data

🤔Before reading on: do you think Pearson correlation scales easily to many variables or large datasets? Commit to your answer.

Concept: In big data or many variables, Pearson correlation is used in matrices and can be computationally expensive or misleading without preprocessing.

Correlation matrices show pairwise correlations among many variables. In large datasets, noise and multiple testing require corrections. Dimensionality reduction or feature selection often follows correlation analysis.

Result

You understand how Pearson correlation fits into complex data workflows and its computational challenges.

Recognizing scaling challenges and noise effects is key for expert data analysis using correlation.

Under the Hood

Pearson correlation calculates the covariance of two variables normalized by their standard deviations. Covariance measures how two variables vary together. Dividing by standard deviations scales this to a fixed range between -1 and 1, making it easier to interpret regardless of units or scale.

Why designed this way?

The formula was designed to provide a standardized measure of linear association that is unitless and comparable across different datasets. Alternatives like covariance alone depend on units and scale, making interpretation difficult. The normalization allows consistent interpretation.

Data pairs (x_i, y_i)
      │
      ▼
Calculate means (mean_x, mean_y)
      │
      ▼
Calculate deviations (x_i - mean_x), (y_i - mean_y)
      │
      ▼
Multiply deviations and sum over all pairs
      │
      ▼
Divide by (n-1) to get covariance
      │
      ▼
Divide covariance by (std_x * std_y) to normalize
      │
      ▼
Pearson correlation coefficient (r) between -1 and 1

Myth Busters - 3 Common Misconceptions

Quick: Does a Pearson correlation of 0 mean the variables are completely unrelated? Commit yes or no.

Common Belief: A Pearson correlation of 0 means no relationship at all between variables.

Reality: A correlation of 0 means no linear relationship, but variables can still have a strong non-linear relationship.

Why it matters: Assuming zero correlation means no relationship can cause missing important patterns, like curved or complex associations.

Quick: Does a high Pearson correlation prove one variable causes the other? Commit yes or no.

Common Belief: A high Pearson correlation proves that one variable causes changes in the other.

Reality: Correlation does not imply causation; two variables can be correlated due to coincidence or a third factor.

Why it matters: Mistaking correlation for causation can lead to wrong decisions, like assuming ice cream sales cause shark attacks.

Quick: Can Pearson correlation handle data with many outliers well? Commit yes or no.

Common Belief: Pearson correlation is robust and works well even if data has many outliers.

Reality: Outliers can greatly distort Pearson correlation, making it unreliable without cleaning or using robust methods.

Why it matters: Ignoring outliers can produce misleading correlation results, causing wrong interpretations.

Expert Zone

1

Pearson correlation is sensitive to sample size; small samples can produce unstable estimates that appear strong or weak by chance.

2

In multivariate data, interpreting pairwise correlations without considering confounding variables can be misleading; partial correlation helps control for this.

3

The p-value from scipy’s pearsonr assumes data is normally distributed; violations can affect significance testing accuracy.

When NOT to use

Avoid Pearson correlation when data relationships are non-linear, contain many outliers, or variables are ordinal or categorical. Use Spearman or Kendall rank correlations or robust correlation measures instead.

Production Patterns

Professionals use Pearson correlation to explore initial data relationships, create correlation matrices for feature selection, and validate assumptions before regression. In finance, it helps measure asset co-movements; in biology, gene expression relationships.

Connections

Covariance

Pearson correlation is a normalized form of covariance.

Understanding covariance helps grasp why Pearson correlation standardizes it to a fixed range for easier interpretation.

Linear regression

Pearson correlation measures strength of linear relationship, which linear regression models explicitly.

Knowing correlation helps understand the fit quality and direction in regression analysis.

Signal processing

Correlation in signal processing measures similarity between signals, similar to Pearson correlation measuring linear association.

Recognizing correlation as a similarity measure across fields deepens understanding of its broad applications.

Common Pitfalls

#1 Using Pearson correlation on data with strong outliers.

Wrong approach:

import scipy.stats as stats
x = [1, 2, 3, 4, 100]
y = [2, 4, 6, 8, 10]
corr, p = stats.pearsonr(x, y)
print(corr)

Correct approach:

import scipy.stats as stats
x = [1, 2, 3, 4, 100]
y = [2, 4, 6, 8, 10]
# Use Spearman correlation to reduce outlier effect
corr, p = stats.spearmanr(x, y)
print(corr)

Root cause:

Pearson correlation is sensitive to extreme values, which distort the linear relationship measure.

#2 Interpreting correlation as causation.

Wrong approach:

print("High correlation means X causes Y")

Correct approach:

print("Correlation shows association, but further analysis is needed to prove causation")

Root cause:

Confusing association with cause leads to incorrect conclusions.

#3 Applying Pearson correlation to categorical data.

Wrong approach:

import scipy.stats as stats
x = ['red', 'blue', 'green']
y = ['small', 'medium', 'large']
corr, p = stats.pearsonr(x, y)

Correct approach:

print("Pearson correlation requires numeric data; use other methods for categorical data")

Root cause:

Pearson correlation requires numeric inputs; categorical data must be encoded or analyzed differently.

Key Takeaways

Pearson correlation quantifies the strength and direction of a linear relationship between two numeric variables.

It produces a value between -1 and 1, where values near ±1 indicate strong linear association and values near 0 indicate weak or no linear association.

Pearson correlation assumes linearity, normal distribution, and no extreme outliers; violating these can mislead results.

Using scipy’s pearsonr function provides both the correlation coefficient and a p-value to assess statistical significance.

Correlation does not imply causation; it only measures association, so careful interpretation is essential.