Outlier detection with IQR in Pandas - Step-by-Step Execution

Concept Flow - Outlier detection with IQR

Calculate Q1 (25th percentile)

↓

Calculate Q3 (75th percentile)

↓

Compute IQR = Q3 - Q1

↓

Determine lower bound = Q1 - 1.5*IQR

↓

Determine upper bound = Q3 + 1.5*IQR

↓

Flag data points < lower bound or > upper bound as outliers

↓

Done

Step-by-step, we find the middle 50% range of data, then mark points far outside this range as outliers.

Execution Sample

Pandas

import pandas as pd

data = pd.Series([10, 12, 14, 15, 18, 19, 20, 100])
Q1 = data.quantile(0.25)
Q3 = data.quantile(0.75)
IQR = Q3 - Q1
outliers = data[(data < Q1 - 1.5*IQR) | (data > Q3 + 1.5*IQR)]

This code finds outliers in a list of numbers using the IQR method.

Execution Table

Step	Action	Value/Calculation	Result
1	Calculate Q1 (25th percentile)	data.quantile(0.25)	13.25
2	Calculate Q3 (75th percentile)	data.quantile(0.75)	19.25
3	Compute IQR	Q3 - Q1	6.0
4	Calculate lower bound	Q1 - 1.5 * IQR	13.25 - 9.0 = 4.25
5	Calculate upper bound	Q3 + 1.5 * IQR	19.25 + 9.0 = 28.25
6	Identify outliers	Values < 4.25 or > 28.25	Only 100 is > 28.25, so outlier
7	Output outliers	data[(data < 4.25) | (data > 28.25)]	100
8	End	No more steps	Outlier detection complete

💡 All data points checked; only 100 lies outside bounds, marked as outlier.

Variable Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4	After Step 5	After Step 6	Final
data	[10,12,14,15,18,19,20,100]	[10,12,14,15,18,19,20,100]	[10,12,14,15,18,19,20,100]	[10,12,14,15,18,19,20,100]	[10,12,14,15,18,19,20,100]	[10,12,14,15,18,19,20,100]	[10,12,14,15,18,19,20,100]	[10,12,14,15,18,19,20,100]
Q1	N/A	13.25	13.25	13.25	13.25	13.25	13.25	13.25
Q3	N/A	N/A	19.25	19.25	19.25	19.25	19.25	19.25
IQR	N/A	N/A	N/A	6.0	6.0	6.0	6.0	6.0
lower_bound	N/A	N/A	N/A	N/A	4.25	4.25	4.25	4.25
upper_bound	N/A	N/A	N/A	N/A	N/A	28.25	28.25	28.25
outliers	N/A	N/A	N/A	N/A	N/A	N/A	[100]	[100]

Key Moments - 3 Insights

Why do we multiply IQR by 1.5 to find bounds?

Multiplying IQR by 1.5 sets a range beyond the middle 50% to catch unusually far points. See execution_table steps 4 and 5 where bounds are calculated using 1.5*IQR.

Why is 100 an outlier but 20 is not, even though 20 is far from most data?

Because 20 lies within the upper bound (28.25), it is not flagged. Only values beyond 28.25 are outliers, as shown in execution_table step 6.

Can outliers be below the lower bound?

Yes, any value less than lower bound (4.25 here) is also an outlier. In this data, no values are below 4.25, so no low outliers appear.

Visual Quiz - 3 Questions

Look at the execution_table at step 3. What is the IQR value?

A 13.25

B 19.25

C 6.0

D 28.25

💡 Hint

Check the 'Result' column at step 3 in execution_table.

At which step do we identify the outlier values?

A Step 6

B Step 4

C Step 2

D Step 8

💡 Hint

Look for the step where values outside bounds are found in execution_table.

If the data had a value 3 instead of 10, what would happen to the lower bound?

A Lower bound would increase

B Lower bound would decrease

C Lower bound stays the same

D Lower bound becomes zero

💡 Hint

Think about how Q1 changes if data has smaller values; check variable_tracker for Q1 and lower_bound.

Concept Snapshot

Outlier detection with IQR:
1. Calculate Q1 (25th percentile) and Q3 (75th percentile).
2. Compute IQR = Q3 - Q1.
3. Define lower bound = Q1 - 1.5*IQR and upper bound = Q3 + 1.5*IQR.
4. Data points outside these bounds are outliers.
Use pandas quantile() and boolean indexing to find outliers.

Full Transcript

Outlier detection with IQR involves finding the middle 50% range of data between Q1 and Q3. We calculate the interquartile range (IQR) by subtracting Q1 from Q3. Then, we set lower and upper bounds by subtracting and adding 1.5 times the IQR to Q1 and Q3 respectively. Any data points outside these bounds are considered outliers. In the example, the data series has values including 100, which lies beyond the upper bound and is flagged as an outlier. This method helps identify unusually high or low values compared to the bulk of data.

Practice

1. What does the Interquartile Range (IQR) represent in a dataset?

easy

A. The range between the 25th percentile (Q1) and the 75th percentile (Q3)

B. The difference between the maximum and minimum values

C. The average value of the dataset

D. The total number of data points

Solution

1. Step 1: Understand quartiles

   Quartiles divide data into four equal parts. Q1 is the 25th percentile, Q3 is the 75th percentile.

2. Step 2: Define IQR

   IQR is the difference between Q3 and Q1, showing the middle 50% spread of data.

3. Final Answer:

   The range between the 25th percentile (Q1) and the 75th percentile (Q3) -> Option A

4. Quick Check:

   IQR = Q3 - Q1 [OK]

Hint: IQR = Q3 minus Q1, middle half range [OK]

Common Mistakes:

• Confusing IQR with total range
• Thinking IQR is the average
• Mixing up quartiles with percentiles

2. Which of the following pandas code snippets correctly calculates the IQR of a DataFrame column df['score']?

easy

A. iqr = df['score'].mean() - df['score'].median()

B. iqr = df['score'].max() - df['score'].min()

C. iqr = df['score'].quantile(0.75) - df['score'].quantile(0.25)

D. iqr = df['score'].std()

Solution

1. Step 1: Recall IQR formula

   IQR is Q3 minus Q1, where Q3 is 75th percentile and Q1 is 25th percentile.

2. Step 2: Check pandas quantile usage

   Using quantile(0.75) and quantile(0.25) correctly gets Q3 and Q1.

3. Final Answer:

   iqr = df['score'].quantile(0.75) - df['score'].quantile(0.25) -> Option C

4. Quick Check:

   Use quantile(0.75) - quantile(0.25) for IQR [OK]

Hint: Use quantile(0.75) minus quantile(0.25) for IQR [OK]

Common Mistakes:

• Using mean or median instead of quantiles
• Using max-min instead of IQR
• Using standard deviation instead of IQR

3. Given the following code, what will be the output of outliers?

import pandas as pd

data = {'values': [10, 12, 14, 15, 18, 100, 105]}
df = pd.DataFrame(data)
Q1 = df['values'].quantile(0.25)
Q3 = df['values'].quantile(0.75)
IQR = Q3 - Q1
lower_bound = Q1 - 1.5 * IQR
upper_bound = Q3 + 1.5 * IQR
outliers = df[(df['values'] < lower_bound) | (df['values'] > upper_bound)]
print(outliers)

medium

A. Rows with values 10 and 12

B. Rows with values 100 and 105

C. All rows because 1.5*IQR is too small

D. Empty DataFrame, no outliers detected

Solution

1. Step 1: Calculate Q1, Q3, and IQR

   Q1 = 12.0 (between 12 and 14), Q3 = 18.0 (between 18 and 100), IQR = 6.0.

2. Step 2: Calculate bounds and find outliers

   Lower bound = 12 - 1.5*6 = 3; Upper bound = 18 + 1.5*6 = 27. Values 100 and 105 are above upper bound, so they are outliers.

3. Final Answer:

   Rows with values 100 and 105 -> Option B

4. Quick Check:

   Outliers > Q3 + 1.5*IQR or < Q1 - 1.5*IQR: 100, 105 [OK]

Hint: Outliers lie beyond Q1-1.5*IQR or Q3+1.5*IQR [OK]

Common Mistakes:

• Miscomputing Q1 or Q3
• Using wrong inequality signs
• Ignoring both lower and upper bounds

4. Identify the error in this code snippet for detecting outliers using IQR:

Q1 = df['data'].quantile(0.25)
Q3 = df['data'].quantile(0.75)
IQR = Q3 - Q1
outliers = df[(df['data'] > Q3 + 1.5 * IQR) & (df['data'] < Q1 - 1.5 * IQR)]

medium

A. No error, code is correct

B. IQR calculation is incorrect

C. Quantile values are swapped

D. The condition uses '&' instead of '|' to combine bounds

Solution

1. Step 1: Understand outlier condition

   Outliers are values less than lower bound OR greater than upper bound, so conditions should be combined with OR (|).

2. Step 2: Check code condition

   The code uses AND (&), which means no value can be both greater than upper bound and less than lower bound simultaneously.

3. Final Answer:

   The condition uses '&' instead of '|' to combine bounds -> Option D

4. Quick Check:

   Use '|' for outlier condition, not '&' [OK]

Hint: Use OR (|) to combine outlier conditions, not AND (&) [OK]

Common Mistakes:

• Using AND instead of OR for outlier filtering
• Swapping Q1 and Q3 values
• Calculating IQR incorrectly

5. You have a DataFrame df with a column 'height'. You want to remove outliers using IQR but keep the original index. Which code correctly filters out outliers and keeps the index intact?

hard

A. Q1 = df['height'].quantile(0.25)
Q3 = df['height'].quantile(0.75)
IQR = Q3 - Q1
filtered_df = df[(df['height'] >= Q1 - 1.5*IQR) & (df['height'] <= Q3 + 1.5*IQR)]

B. Q1 = df['height'].quantile(0.25)
Q3 = df['height'].quantile(0.75)
IQR = Q3 - Q1
filtered_df = df.reset_index(drop=True)[(df['height'] >= Q1 - 1.5*IQR) & (df['height'] <= Q3 + 1.5*IQR)]

C. Q1 = df['height'].quantile(0.25)
Q3 = df['height'].quantile(0.75)
IQR = Q3 - Q1
filtered_df = df[(df['height'] > Q1 + 1.5*IQR) | (df['height'] < Q3 - 1.5*IQR)]

D. Q1 = df['height'].quantile(0.25)
Q3 = df['height'].quantile(0.75)
IQR = Q3 - Q1
filtered_df = df[(df['height'] < Q1 - 1.5*IQR) & (df['height'] > Q3 + 1.5*IQR)]

Solution

1. Step 1: Calculate IQR and bounds correctly

   Calculate Q1, Q3, and IQR. Define lower bound as Q1 - 1.5*IQR and upper bound as Q3 + 1.5*IQR.

2. Step 2: Filter rows within bounds and keep index

   Use boolean indexing with AND (&) to keep rows inside bounds. Do not reset index to keep original indices.

3. Final Answer:

   Code in option A correctly filters and keeps index -> Option A

4. Quick Check:

   Filter with AND inside bounds, no reset_index [OK]

Hint: Filter with AND inside bounds; avoid resetting index [OK]

Common Mistakes:

• Using OR instead of AND to filter
• Resetting index and losing original row labels
• Swapping bounds or using wrong inequalities