Percentiles and quantiles in SciPy - Deep Dive

Overview - Percentiles and quantiles

What is it?

Percentiles and quantiles are ways to divide data into parts to understand its distribution. A percentile tells you the value below which a certain percent of data falls. Quantiles split data into equal-sized groups, like quarters or tenths. These help summarize and compare data easily.

Why it matters

Without percentiles and quantiles, it would be hard to grasp how data spreads or where values stand compared to others. They help in making decisions, like knowing if a student's score is high or low compared to peers. In real life, they guide things like setting income brackets or understanding test results.

Where it fits

Before learning percentiles and quantiles, you should know basic statistics like mean, median, and sorting data. After this, you can explore advanced data summaries, box plots, and statistical tests that use these concepts.

Mental Model

Core Idea

Percentiles and quantiles split data into parts to show how values compare within the whole set.

Think of it like...

Imagine a race where runners line up from fastest to slowest. Percentiles tell you the runner who finished faster than a certain percent of others, like the top 10%. Quantiles divide the runners into equal groups, like splitting them into four teams based on finish order.

Data sorted: 1 3 5 7 9 11 13 15 17 19

Percentiles:
  10th percentile -> value below which 10% of data lies
  50th percentile (median) -> middle value
  90th percentile -> value below which 90% of data lies

Quantiles:
  Quartiles (4 groups): Q1 | Q2 | Q3
  Each group has 25% of data

┌───────────────┬───────────────┬───────────────┬───────────────┐
│  0-25% (Q1)  │ 25-50% (Q2)  │ 50-75% (Q3)  │ 75-100% (Q4) │
└───────────────┴───────────────┴───────────────┴───────────────┘

Build-Up - 6 Steps

1

Foundation

Understanding sorted data basics

Concept: Data must be sorted to find percentiles and quantiles.

Take a list of numbers and arrange them from smallest to largest. This order helps us find positions that split the data into parts.

Result

Sorted data allows us to pick values at specific positions representing percentiles or quantiles.

Knowing that sorting is the first step clarifies why percentiles and quantiles depend on data order, not just values.

2

Foundation

Defining percentiles simply

Concept: A percentile shows the value below which a certain percent of data falls.

For example, the 25th percentile is the value below which 25% of data points lie. The 50th percentile is the median, splitting data in half.

Result

You can say, '70% of data is below this value' using percentiles.

Percentiles give a clear way to understand data spread and position without looking at every number.

3

Intermediate

Quantiles as equal data splits

🤔Before reading on: Do you think quartiles always split data into exactly equal groups? Commit to your answer.

Concept: Quantiles divide data into equal-sized groups, like halves, quarters, or tenths.

Quartiles split data into 4 groups, each with 25% of data. Quintiles split into 5 groups, each 20%. These help summarize data distribution.

Result

Quantiles provide multiple cut points that divide data evenly, useful for comparisons.

Understanding quantiles as equal splits helps in grasping how data is segmented beyond just single points.

4

Intermediate

Calculating percentiles with scipy

🤔Before reading on: Do you think scipy's percentile function returns the exact data value or an interpolated value? Commit to your answer.

Concept: Scipy provides tools to calculate percentiles, handling interpolation when needed.

Using scipy.stats.percentileofscore or numpy.percentile, you input data and desired percentile. The function returns the value at that percentile, interpolating if the exact position is between data points. Example: import numpy as np data = np.array([1,3,5,7,9]) np.percentile(data, 40) # returns 3.6 (interpolated between 3 and 5)

Result

You get precise percentile values even with small or uneven data sets.

Knowing interpolation is used prevents confusion when percentile values don't match exact data points.

5

Advanced

Different interpolation methods in scipy

🤔Before reading on: Do you think changing interpolation methods affects percentile results? Commit to your answer.

Concept: Scipy allows choosing how to interpolate between data points for percentile calculation.

Methods include 'linear', 'lower', 'higher', 'nearest', and 'midpoint'. Each changes how the percentile value is picked when it falls between two data points. Example: np.percentile(data, 40, interpolation='lower') # returns 3 np.percentile(data, 40, interpolation='higher') # returns 5

Result

You can control percentile calculation to fit your analysis needs.

Understanding interpolation options helps tailor percentile calculations for accuracy or conservatism.

6

Expert

Handling edge cases and ties in percentiles

🤔Before reading on: Do you think ties in data affect percentile calculations? Commit to your answer.

Concept: When data has repeated values or small size, percentile calculations can behave unexpectedly.

Ties mean multiple data points have the same value, which can cause percentile positions to be ambiguous. Scipy's interpolation methods and ranking rules handle these cases differently. Also, very small datasets may produce less stable percentile estimates. Example: data = np.array([1,2,2,2,3]) np.percentile(data, 50) # median might be 2, but interpolation affects exact value

Result

Percentile results remain consistent and meaningful even with tricky data.

Knowing how ties and small samples affect percentiles prevents misinterpretation of results in real data.

Under the Hood

Percentile calculation sorts data and finds the rank position corresponding to the desired percentile. If this position is not an integer, interpolation estimates the value between neighboring data points. Scipy implements several interpolation methods to handle this smoothly. Internally, it uses efficient sorting and indexing algorithms to handle large datasets quickly.

Why designed this way?

Percentiles needed a standard way to summarize data distribution beyond simple averages. Interpolation methods were introduced to handle real-world data that rarely fits exact percentile positions. Scipy's flexible design allows users to choose interpolation based on their analysis goals, balancing precision and robustness.

Data array (unsorted): [7, 1, 9, 3, 5]
          ↓ sort
Sorted data: [1, 3, 5, 7, 9]

Percentile rank calculation:
Desired percentile: p%
Position = (p/100) * (N - 1)

If position is integer:
  value = data[position]
Else:
  interpolate between data[floor(position)] and data[ceil(position)]

┌───────────────┐
│   Input data  │
└──────┬────────┘
       ↓ sort
┌───────────────┐
│ Sorted data   │
└──────┬────────┘
       ↓ calculate position
┌───────────────┐
│ Position in   │
│ sorted data   │
└──────┬────────┘
       ↓ interpolate if needed
┌───────────────┐
│ Percentile    │
│ value output  │
└───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does the 50th percentile always equal the median? Commit to yes or no.

Common Belief: The 50th percentile is always the median value in the data.

Reality: While often close, the 50th percentile may differ from the median due to interpolation methods, especially in small or uneven datasets.

Why it matters: Assuming they are always equal can lead to incorrect conclusions about data center and spread.

Quick: Do quantiles always split data into groups with exactly equal counts? Commit to yes or no.

Common Belief: Quantiles always divide data into groups with exactly the same number of data points.

Reality: When data size isn't divisible evenly, quantiles approximate equal groups, sometimes with slightly different sizes.

Why it matters: Expecting perfect splits can cause confusion when group sizes differ slightly in reports or visualizations.

Quick: Does changing interpolation method in percentile calculation not affect results much? Commit to yes or no.

Common Belief: Interpolation method choice has little impact on percentile values.

Reality: Interpolation method can significantly change percentile results, especially for small datasets or percentiles near data points.

Why it matters: Ignoring interpolation effects can cause inconsistent or misleading analysis outcomes.

Quick: Is percentile calculation always meaningful for very small datasets? Commit to yes or no.

Common Belief: Percentiles are always reliable regardless of dataset size.

Reality: With very small datasets, percentile estimates can be unstable or misleading due to limited data points.

Why it matters: Using percentiles blindly on small data can lead to false confidence in results.

Expert Zone

1

Percentile calculation methods differ across software; knowing scipy's approach avoids cross-tool confusion.

2

Interpolation choice affects statistical tests and confidence intervals that rely on percentiles.

3

Handling ties properly is crucial in ranking-based analyses like non-parametric tests.

When NOT to use

Percentiles and quantiles are less useful for categorical data or very small datasets where exact ranks are unstable. Alternatives include mode for categories or bootstrapping for small samples.

Production Patterns

In production, percentiles are used for performance monitoring (e.g., 95th percentile latency), risk assessment, and customer segmentation. Choosing interpolation and handling ties carefully ensures reliable automated reports.

Connections

Box plots

Builds-on

Box plots visually summarize data distribution using quartiles, a type of quantile, making percentiles tangible.

Cumulative distribution function (CDF)

Same pattern

Percentiles correspond to points on the CDF, linking data ranking to probability concepts.

Income tax brackets (Economics)

Application analogy

Tax brackets use quantiles to group incomes, showing how data science concepts apply in real-world policy.

Common Pitfalls

#1 Using percentile calculation on unsorted data.

Wrong approach:

import numpy as np

data = np.array([5, 1, 9, 3, 7])
np.percentile(data, 50) # assuming data is unsorted manually

Correct approach:

import numpy as np

data = np.array([5, 1, 9, 3, 7])
data_sorted = np.sort(data)
np.percentile(data_sorted, 50)

Root cause:

Misunderstanding that percentile functions sort data internally or expecting manual sorting is unnecessary.

#2 Ignoring interpolation method leading to unexpected percentile values.

Wrong approach:

np.percentile(data, 40) # default interpolation without consideration

Correct approach:

np.percentile(data, 40, interpolation='linear') # explicitly choosing interpolation

Root cause:

Not knowing interpolation affects results, causing confusion when values differ from expectations.

#3 Applying percentiles to very small datasets without caution.

Wrong approach:

data = np.array([1, 2])
np.percentile(data, 90) # trusting result blindly

Correct approach:

data = np.array([1, 2])
# Use caution or alternative methods like bootstrapping for small data

Root cause:

Assuming percentile calculations are always stable regardless of data size.

Key Takeaways

Percentiles and quantiles help divide data into parts to understand its distribution and relative positions.

Data must be sorted before calculating percentiles or quantiles, as order determines these values.

Interpolation methods in scipy affect percentile results, especially when exact positions fall between data points.

Ties and small datasets require careful handling to avoid misleading percentile calculations.

These concepts are widely used in real-world analysis, from performance metrics to economic policies.

Practice

1. What does the 90th percentile represent in a dataset?

easy

A. The average of the top 90% of data

B. The value below which 90% of the data falls

C. The value above which 90% of the data falls

D. The middle value of the dataset

Solution

1. Step 1: Understand percentile definition

   A percentile indicates the value below which a given percentage of observations fall.

2. Step 2: Apply to 90th percentile

   The 90th percentile means 90% of data points are below this value.

3. Final Answer:

   The value below which 90% of the data falls -> Option B

4. Quick Check:

   Percentile = value below % data [OK]

Hint: Percentile = value below that percent of data [OK]

Common Mistakes:

• Confusing percentile with average
• Thinking percentile is value above percentage
• Mixing percentile with median

2. Which of the following is the correct way to find the 25th percentile using scipy.stats?

easy

A. stats.quantile(data, 25)

B. stats.percentile(data, 0.25)

C. stats.mquantiles(data, prob=0.25)

D. stats.scoreatpercentile(data, 25)

Solution

1. Step 1: Recall scipy function for percentile

   The function to find percentile is stats.scoreatpercentile with arguments (data, percentile).

2. Step 2: Check options

   stats.scoreatpercentile(data, 25) uses stats.scoreatpercentile(data, 25), which is correct for 25th percentile.

3. Final Answer:

   stats.scoreatpercentile(data, 25) -> Option D

4. Quick Check:

   Use scoreatpercentile for percentiles [OK]

Hint: Use stats.scoreatpercentile(data, percentile) [OK]

Common Mistakes:

• Using wrong function names like percentile or quantile
• Passing 0.25 instead of 25 for percentile
• Confusing quantiles with percentiles

3. Given the data array [10, 20, 30, 40, 50], what is the output of stats.scoreatpercentile(data, 40)?

medium

A. 26.0

B. 22.0

C. 30.0

D. 20.0

Solution

1. Step 1: Understand percentile calculation

   40th percentile means the value below which 40% of data lies. Data sorted: [10, 20, 30, 40, 50].

2. Step 2: Calculate 40th percentile using interpolation

   Position = (40/100)*(n-1) = 0.4*4 = 1.6. Interpolate between data[1]=20 and data[2]=30: 20 + 0.6*(30-20) = 20 + 6 = 26.0.

3. Final Answer:

   26.0 -> Option A

4. Quick Check:

   Interpolation gives 26.0 [OK]

Hint: Percentile position = p/100*(n-1), interpolate values [OK]

Common Mistakes:

• Choosing nearest data point without interpolation
• Using index p instead of p/100*(n-1)
• Confusing percentile with median

4. The code below is intended to find the quartiles of data using scipy but raises an error. What is the error?

import scipy.stats as stats
data = [5, 10, 15, 20, 25]
quartiles = stats.mstats.mquantiles(data, prob=[0.25, 0.5, 0.75])
print(quartiles)

medium

A. mquantiles requires numpy array, not list

B. prob argument should be a single float, not a list

C. No error, code runs correctly

D. mquantiles is not in scipy.stats.mstats

Solution

1. Step 1: Check function location and usage

   stats.mstats.mquantiles is a valid function and accepts prob as a list for multiple quantiles.

2. Step 2: Check input data type

   mquantiles accepts list or array-like data, so list input is valid.

3. Final Answer:

   No error, code runs correctly -> Option C

4. Quick Check:

   mquantiles accepts list and prob list [OK]

Hint: mquantiles accepts list and prob list [OK]

Common Mistakes:

• Assuming mquantiles only accepts numpy arrays
• Thinking prob must be a single float
• Confusing function location in scipy

5. You have a dataset of exam scores: [55, 70, 65, 80, 90, 85, 75]. You want to divide the scores into four equal groups (quartiles) using scipy. Which code snippet correctly computes the quartiles?

hard

A. stats.mstats.mquantiles(scores, prob=[0.25, 0.5, 0.75])

B. stats.scoreatpercentile(scores, [25, 50, 75])

C. np.percentile(scores, [0.25, 0.5, 0.75])

D. stats.mstats.mquantiles(scores, prob=[25, 50, 75])

Solution

1. Step 1: Identify correct function for quartiles

   Quartiles are quantiles at 25%, 50%, and 75%. stats.mstats.mquantiles accepts prob as fractions between 0 and 1.

2. Step 2: Check options for correct prob format

   stats.mstats.mquantiles(scores, prob=[0.25, 0.5, 0.75]) uses prob=[0.25, 0.5, 0.75], which is correct. stats.scoreatpercentile(scores, [25, 50, 75]) uses scoreatpercentile but with a list, which is invalid. np.percentile(scores, [0.25, 0.5, 0.75]) uses numpy but with fractions instead of percentages. stats.mstats.mquantiles(scores, prob=[25, 50, 75]) uses prob with values >1, which is invalid.

3. Final Answer:

   stats.mstats.mquantiles(scores, prob=[0.25, 0.5, 0.75]) -> Option A

4. Quick Check:

   mquantiles prob values are fractions [OK]

Hint: Use mquantiles with prob as fractions (0.25, 0.5, 0.75) [OK]

Common Mistakes:

• Passing percent values instead of fractions to prob
• Using scoreatpercentile with list of percentiles
• Confusing numpy percentile input format