Boolean filtering in Data Analysis Python - Step-by-Step Execution

Concept Flow - Boolean filtering

Start with DataFrame

↓

Create Boolean Condition

↓

Apply Condition to DataFrame

↓

Get Filtered DataFrame

↓

End

We start with a data table, create a true/false condition for each row, then keep only rows where the condition is true.

Execution Sample

Data Analysis Python

import pandas as pd

df = pd.DataFrame({'Age': [18, 22, 15, 30]})
filtered = df[df['Age'] >= 18]
print(filtered)

This code keeps only rows where Age is 18 or more.

Execution Table

Step	DataFrame Rows	Condition (Age >= 18)	Filtered Rows
1	[18, 22, 15, 30]	[True, True, False, True]	[18, 22, 30]
2	Filtered DataFrame ready	N/A	Rows with True condition kept

💡 Filtering done, only rows where Age >= 18 remain

Variable Tracker

Variable	Start	After Filtering	Final
df['Age']	[18, 22, 15, 30]	[18, 22, 15, 30]	[18, 22, 15, 30]
condition	N/A	[True, True, False, True]	[True, True, False, True]
filtered	N/A	[18, 22, 30]	[18, 22, 30]

Key Moments - 2 Insights

Why does the filtered DataFrame only have 3 rows instead of 4?

Because the condition 'Age >= 18' is False for the row with Age 15, so that row is excluded (see execution_table step 1).

What type of values does the condition produce?

It produces a list of True/False values, one for each row, indicating if the row meets the condition (see 'Condition' column in execution_table).

Visual Quiz - 3 Questions

Look at the execution table, what is the condition value for the row where Age is 15?

A True

B False

C None

D Error

💡 Hint

Check the 'Condition (Age >= 18)' column in execution_table at step 1

At which step do we get the filtered DataFrame with only rows where Age >= 18?

A Step 1

B Step 2

C Before Step 1

D No step

💡 Hint

Look at the 'Filtered Rows' column in execution_table

If we change the condition to 'Age > 20', how would the filtered rows change?

A [15, 18, 22, 30]

B [18, 22, 30]

C [22, 30]

D [15]

💡 Hint

Think about which ages are strictly greater than 20

Concept Snapshot

Boolean filtering in pandas:
- Create a condition that returns True/False per row
- Use df[condition] to keep only True rows
- Result is a smaller DataFrame
- Useful to select data by criteria
- Condition can be any comparison or logical expression

Full Transcript

Boolean filtering means selecting rows from a data table where a condition is true. We start with a DataFrame, create a condition like 'Age >= 18' that returns True or False for each row. Then we use this condition to keep only rows where it is True. For example, if ages are 18, 22, 15, and 30, only rows with 18, 22, and 30 remain after filtering. The condition is a list of True/False values matching each row. This method helps us focus on data that meets our criteria.

Practice

1. What does boolean filtering do in data analysis with Python?

easy

A. Changes the data type of columns

B. Removes duplicate rows from the data

C. Sorts the data in ascending order

D. Selects rows where a condition is true

Solution

1. Step 1: Understand boolean filtering purpose

   Boolean filtering is used to pick rows that meet a true condition.

2. Step 2: Compare options to definition

   Only Selects rows where a condition is true describes selecting rows based on true conditions.

3. Final Answer:

   Selects rows where a condition is true -> Option D

4. Quick Check:

   Boolean filtering = Select rows true condition [OK]

Hint: Boolean filtering picks rows where condition is true [OK]

Common Mistakes:

• Confusing filtering with sorting
• Thinking it changes data types
• Assuming it removes duplicates

2. Which of the following is the correct syntax to filter rows where column age is greater than 30 in a DataFrame df?

easy

A. df['age' > 30]

B. df[df['age'] > 30]

C. df.age[df.age > 30]

D. df[df['age'] < 30]

Solution

1. Step 1: Identify correct boolean filtering syntax

   The correct way is to put the condition inside square brackets after the DataFrame.

2. Step 2: Check each option

   df[df['age'] > 30] uses df[df['age'] > 30], which is correct syntax. df['age' > 30] is invalid syntax. df.age[df.age > 30] incorrectly filters only the age column, returning a Series not DataFrame. df[df['age'] < 30] filters less than 30, not greater.

3. Final Answer:

   df[df['age'] > 30] -> Option B

4. Quick Check:

   Correct syntax = df[df['age'] > 30] [OK]

Hint: Use df[condition] with column in quotes for filtering [OK]

Common Mistakes:

• Missing brackets around condition
• Using wrong comparison operator
• Not quoting column name

3. Given the DataFrame df below:

import pandas as pd
df = pd.DataFrame({
  'name': ['Anna', 'Bob', 'Cara', 'Dan'],
  'score': [85, 92, 78, 90],
  'passed': [True, True, False, True]
})

What will df[df['score'] > 80 and df['passed']] return?

medium

A. ValueError due to missing parentheses

B. Rows where score > 80 or passed is true

C. Rows where score > 80 and passed is true

D. Empty DataFrame

Solution

1. Step 1: Analyze the boolean expression

   The expression uses and without parentheses around each condition, which causes a ValueError in pandas.

2. Step 2: Understand pandas boolean filtering rules

   Each condition must be in parentheses when combined with & or |. So df['score'] > 80 and df['passed'] raises ValueError.

3. Final Answer:

   ValueError due to missing parentheses -> Option A

4. Quick Check:

   Use parentheses around conditions with & or | [OK]

Hint: Always wrap each condition in parentheses when using & or | [OK]

Common Mistakes:

• Not using parentheses around conditions
• Confusing & with and
• Expecting output without fixing syntax

4. Consider this code snippet:

import pandas as pd
df = pd.DataFrame({'A': [1, 2, 3, 4]})
filtered = df[df['A'] > 2 or df['A'] < 4]

What is the error and how to fix it?

medium

A. ValueError: Use parentheses around conditions with or

B. SyntaxError: Missing colon after if

C. No error, code runs fine

D. ValueError: Cannot compare integers

Solution

1. Step 1: Identify the error cause

   The expression df['A'] > 2 or df['A'] < 4 lacks parentheses around each condition, causing a ValueError.

2. Step 2: Fix the error by adding parentheses

   Correct syntax is df[(df['A'] > 2) | (df['A'] < 4)] to ensure proper evaluation.

3. Final Answer:

   ValueError: Use parentheses around conditions with or -> Option A

4. Quick Check:

   Parentheses needed around each condition with | or & [OK]

Hint: Wrap each condition in parentheses when using | or & [OK]

Common Mistakes:

• Forgetting parentheses around conditions
• Using Python 'or' instead of |
• Assuming no error with missing parentheses

5. You have a DataFrame df with columns age and income. You want to filter rows where age is between 25 and 40 inclusive, and income is above 50000. Which code correctly applies boolean filtering?

hard

A. df[(df['age'] > 25) | (df['age'] < 40) & (df['income'] > 50000)]

B. df[df['age'] >= 25 & df['age'] <= 40 & df['income'] > 50000]

C. df[(df['age'] >= 25) & (df['age'] <= 40) & (df['income'] > 50000)]

D. df[df['age'] >= 25 and df['age'] <= 40 and df['income'] > 50000]

Solution

1. Step 1: Understand the filtering conditions

   We want age between 25 and 40 inclusive, so age >= 25 and age <= 40, and income > 50000.

2. Step 2: Check correct syntax for combining conditions

   Conditions must be in parentheses and combined with &. df[(df['age'] >= 25) & (df['age'] <= 40) & (df['income'] > 50000)] correctly uses parentheses and & operators.

3. Step 3: Identify errors in other options

   df[df['age'] >= 25 & df['age'] <= 40 & df['income'] > 50000] lacks parentheses around each condition, causing errors. df[(df['age'] > 25) | (df['age'] < 40) & (df['income'] > 50000)] uses | instead of & incorrectly. df[df['age'] >= 25 and df['age'] <= 40 and df['income'] > 50000] uses Python 'and' which does not work with pandas Series.

4. Final Answer:

   df[(df['age'] >= 25) & (df['age'] <= 40) & (df['income'] > 50000)] -> Option C

5. Quick Check:

   Use parentheses and & for multiple conditions [OK]

Hint: Use parentheses and & to combine multiple conditions [OK]

Common Mistakes:

• Using Python 'and' instead of &
• Missing parentheses around conditions
• Mixing & and | incorrectly