Percentiles and quantiles in SciPy - Step-by-Step Execution

Concept Flow - Percentiles and quantiles

Start with data array

↓

Sort data in ascending order

↓

Choose percentile or quantile value

↓

Calculate position in sorted data

↓

Interpolate if needed

↓

Return value at position

↓

End

We start with data, sort it, pick a percentile or quantile, find its position, interpolate if needed, and return the value.

Execution Sample

SciPy

import numpy as np
from scipy import stats

data = np.array([10, 20, 30, 40, 50])
percentile_40 = np.percentile(data, 40)
quantile_0_4 = np.quantile(data, 0.4)

Calculate the 40th percentile and 0.4 quantile of a small data array.

Execution Table

Step	Action	Data State	Calculation	Result
1	Start with data array	[10, 20, 30, 40, 50]	N/A	N/A
2	Sort data	[10, 20, 30, 40, 50]	Already sorted	[10, 20, 30, 40, 50]
3	Calculate 40th percentile position	N/A	pos = (40/100)*(5-1) = 1.6	Position 1.6
4	Interpolate between index 1 and 2	Values at 1=20, 2=30	20 + 0.6*(30-20)	26.0
5	Calculate 0.4 quantile position	N/A	pos = 0.4*(5-1) = 1.6	Position 1.6
6	Interpolate between index 1 and 2	Values at 1=20, 2=30	20 + 0.6*(30-20)	26.0
7	Return results	N/A	N/A	percentile_40 = 26.0, quantile_0_4 = 26.0
8	End	N/A	N/A	Calculation complete

💡 Finished calculating requested percentile and quantile values.

Variable Tracker

Variable	Start	After Step 3	After Step 4	After Step 5	After Step 6	Final
data	[10,20,30,40,50]	[10,20,30,40,50]	[10,20,30,40,50]	[10,20,30,40,50]	[10,20,30,40,50]	[10,20,30,40,50]
percentile_40	N/A	N/A	26.0	26.0	26.0	26.0
quantile_0_4	N/A	N/A	N/A	N/A	26.0	26.0

Key Moments - 2 Insights

Why do we interpolate between two data points for the percentile?

Because the calculated position is not an integer index, we find the value between two points to get a precise percentile value, as shown in steps 3 and 4.

Are percentile and quantile calculations different here?

No, both use the same position calculation and interpolation method, so the 40th percentile and 0.4 quantile give the same result, as seen in steps 3-6.

Visual Quiz - 3 Questions

Look at the execution table, what is the calculated position for the 40th percentile?

A 2.0

B 1.6

C 0.4

D 4.0

💡 Hint

Check Step 3 in the execution table where position is calculated.

At which step do we interpolate between data points?

A Step 2

B Step 6

C Step 4

D Step 7

💡 Hint

Look for where values at indices 1 and 2 are used for interpolation.

If the data array had 10 elements instead of 5, how would the position for the 40th percentile change?

A It would be 3.6

B It would be 4.0

C It would be 1.6

D It would be 0.4

💡 Hint

Position = (percentile/100)*(n-1), so with n=10, calculate accordingly.

Concept Snapshot

Percentiles and quantiles find data values below a given percentage.
Sort data first.
Position = percentile/quantile * (n-1).
Interpolate if position is not whole.
Use numpy.percentile or numpy.quantile for easy calculation.

Full Transcript

We start with a data array and sort it. To find a percentile or quantile, we calculate its position in the sorted data using the formula position = (percentile/100) * (n-1) or position = quantile * (n-1). If the position is not an integer, we interpolate between the two closest data points. For example, the 40th percentile of [10,20,30,40,50] is at position 1.6, so we interpolate between the values at index 1 (20) and index 2 (30), resulting in 26.0. The same applies for the 0.4 quantile. This method gives a precise value representing the percentile or quantile in the data.

Practice

1. What does the 90th percentile represent in a dataset?

easy

A. The average of the top 90% of data

B. The value below which 90% of the data falls

C. The value above which 90% of the data falls

D. The middle value of the dataset

Solution

1. Step 1: Understand percentile definition

   A percentile indicates the value below which a given percentage of observations fall.

2. Step 2: Apply to 90th percentile

   The 90th percentile means 90% of data points are below this value.

3. Final Answer:

   The value below which 90% of the data falls -> Option B

4. Quick Check:

   Percentile = value below % data [OK]

Hint: Percentile = value below that percent of data [OK]

Common Mistakes:

• Confusing percentile with average
• Thinking percentile is value above percentage
• Mixing percentile with median

2. Which of the following is the correct way to find the 25th percentile using scipy.stats?

easy

A. stats.quantile(data, 25)

B. stats.percentile(data, 0.25)

C. stats.mquantiles(data, prob=0.25)

D. stats.scoreatpercentile(data, 25)

Solution

1. Step 1: Recall scipy function for percentile

   The function to find percentile is stats.scoreatpercentile with arguments (data, percentile).

2. Step 2: Check options

   stats.scoreatpercentile(data, 25) uses stats.scoreatpercentile(data, 25), which is correct for 25th percentile.

3. Final Answer:

   stats.scoreatpercentile(data, 25) -> Option D

4. Quick Check:

   Use scoreatpercentile for percentiles [OK]

Hint: Use stats.scoreatpercentile(data, percentile) [OK]

Common Mistakes:

• Using wrong function names like percentile or quantile
• Passing 0.25 instead of 25 for percentile
• Confusing quantiles with percentiles

3. Given the data array [10, 20, 30, 40, 50], what is the output of stats.scoreatpercentile(data, 40)?

medium

A. 26.0

B. 22.0

C. 30.0

D. 20.0

Solution

1. Step 1: Understand percentile calculation

   40th percentile means the value below which 40% of data lies. Data sorted: [10, 20, 30, 40, 50].

2. Step 2: Calculate 40th percentile using interpolation

   Position = (40/100)*(n-1) = 0.4*4 = 1.6. Interpolate between data[1]=20 and data[2]=30: 20 + 0.6*(30-20) = 20 + 6 = 26.0.

3. Final Answer:

   26.0 -> Option A

4. Quick Check:

   Interpolation gives 26.0 [OK]

Hint: Percentile position = p/100*(n-1), interpolate values [OK]

Common Mistakes:

• Choosing nearest data point without interpolation
• Using index p instead of p/100*(n-1)
• Confusing percentile with median

4. The code below is intended to find the quartiles of data using scipy but raises an error. What is the error?

import scipy.stats as stats
data = [5, 10, 15, 20, 25]
quartiles = stats.mstats.mquantiles(data, prob=[0.25, 0.5, 0.75])
print(quartiles)

medium

A. mquantiles requires numpy array, not list

B. prob argument should be a single float, not a list

C. No error, code runs correctly

D. mquantiles is not in scipy.stats.mstats

Solution

1. Step 1: Check function location and usage

   stats.mstats.mquantiles is a valid function and accepts prob as a list for multiple quantiles.

2. Step 2: Check input data type

   mquantiles accepts list or array-like data, so list input is valid.

3. Final Answer:

   No error, code runs correctly -> Option C

4. Quick Check:

   mquantiles accepts list and prob list [OK]

Hint: mquantiles accepts list and prob list [OK]

Common Mistakes:

• Assuming mquantiles only accepts numpy arrays
• Thinking prob must be a single float
• Confusing function location in scipy

5. You have a dataset of exam scores: [55, 70, 65, 80, 90, 85, 75]. You want to divide the scores into four equal groups (quartiles) using scipy. Which code snippet correctly computes the quartiles?

hard

A. stats.mstats.mquantiles(scores, prob=[0.25, 0.5, 0.75])

B. stats.scoreatpercentile(scores, [25, 50, 75])

C. np.percentile(scores, [0.25, 0.5, 0.75])

D. stats.mstats.mquantiles(scores, prob=[25, 50, 75])

Solution

1. Step 1: Identify correct function for quartiles

   Quartiles are quantiles at 25%, 50%, and 75%. stats.mstats.mquantiles accepts prob as fractions between 0 and 1.

2. Step 2: Check options for correct prob format

   stats.mstats.mquantiles(scores, prob=[0.25, 0.5, 0.75]) uses prob=[0.25, 0.5, 0.75], which is correct. stats.scoreatpercentile(scores, [25, 50, 75]) uses scoreatpercentile but with a list, which is invalid. np.percentile(scores, [0.25, 0.5, 0.75]) uses numpy but with fractions instead of percentages. stats.mstats.mquantiles(scores, prob=[25, 50, 75]) uses prob with values >1, which is invalid.

3. Final Answer:

   stats.mstats.mquantiles(scores, prob=[0.25, 0.5, 0.75]) -> Option A

4. Quick Check:

   mquantiles prob values are fractions [OK]

Hint: Use mquantiles with prob as fractions (0.25, 0.5, 0.75) [OK]

Common Mistakes:

• Passing percent values instead of fractions to prob
• Using scoreatpercentile with list of percentiles
• Confusing numpy percentile input format