Boolean indexing in Pandas - Time & Space Complexity

Time Complexity: Boolean indexing

O(n)

Understanding Time Complexity

We want to understand how the time needed to filter data using Boolean indexing changes as the data grows.

How does the filtering time grow when we have more rows in our data?

Scenario Under Consideration

Analyze the time complexity of the following code snippet.

import pandas as pd

df = pd.DataFrame({
    'age': [23, 45, 12, 36, 27],
    'score': [88, 92, 79, 95, 85]
})

filtered = df[df['age'] > 25]

This code creates a table with ages and scores, then selects rows where age is greater than 25.

Identify Repeating Operations

Identify the loops, recursion, array traversals that repeat.

• Primary operation: Checking each row's 'age' value against 25.
• How many times: Once for every row in the table.

How Execution Grows With Input

As the number of rows grows, the time to check each row grows too, roughly in direct proportion.

Input Size (n)	Approx. Operations
10	10 checks
100	100 checks
1000	1000 checks

Pattern observation: Doubling the rows doubles the number of checks needed.

Final Time Complexity

Time Complexity: O(n)

This means the time to filter grows linearly with the number of rows in the data.

Common Mistake

[X] Wrong: "Filtering with Boolean indexing is instant no matter how big the data is."

[OK] Correct: Each row must be checked, so more rows mean more work and more time.

Interview Connect

Understanding how filtering scales helps you write efficient data code and explain your choices clearly.

Self-Check

"What if we filter using multiple conditions combined with && or ||? How would the time complexity change?"

Practice

1. What does boolean indexing in pandas do?

easy

A. Changes the data type of columns

B. Sorts the DataFrame by column values

C. Selects rows based on True/False conditions

D. Merges two DataFrames together

Solution

1. Step 1: Understand boolean indexing

   Boolean indexing uses True/False values to filter rows in a DataFrame.

2. Step 2: Identify its purpose

   It selects only rows where the condition is True, filtering data easily.

3. Final Answer:

   Selects rows based on True/False conditions -> Option C

4. Quick Check:

   Boolean indexing = row filtering [OK]

Hint: Boolean indexing filters rows with True conditions [OK]

Common Mistakes:

• Confusing boolean indexing with sorting
• Thinking it changes data types
• Mixing it up with DataFrame merging

2. Which of the following is the correct syntax to select rows where column 'age' is greater than 30 in a DataFrame df?

easy

A. df['age' > 30]

B. df[df['age'] > 30]

C. df[df.age = 30]

D. df[df['age'] => 30]

Solution

1. Step 1: Check correct boolean indexing syntax

   Use df[condition], where condition is df['age'] > 30.

2. Step 2: Identify syntax errors in other options

   df['age' > 30] misplaces brackets, C uses assignment (=) instead of comparison, D uses invalid operator =>.

3. Final Answer:

   df[df['age'] > 30] -> Option B

4. Quick Check:

   Correct syntax uses df[condition] with comparison [OK]

Hint: Use df[condition] with correct comparison operators [OK]

Common Mistakes:

• Using = instead of > for comparison
• Wrong bracket placement
• Using invalid operators like =>

3. Given the DataFrame df below:

   name  age  score
0  Anna   25     88
1  Bob    32     75
2  Cara   29     95
3  Dave   35     60
4  Ella   28     82

What will df[(df['age'] > 28) & (df['score'] > 70)] return?

medium

A. Rows with age > 28 and score > 70: Anna and Ella

B. Rows with age > 28 and score > 70: Bob and Dave

C. Rows with age > 28 and score > 70: Dave only

D. Rows with age > 28 and score > 70: Bob and Cara

Solution

1. Step 1: Identify rows where age > 28

   Rows: Bob (32), Cara (29), Dave (35) satisfy age > 28.

2. Step 2: From those, select rows with score > 70

   Bob (75), Cara (95), Dave (60). Dave fails score > 70, so only Bob and Cara remain.

3. Final Answer:

   Rows with age > 28 and score > 70: Bob and Cara -> Option D

4. Quick Check:

   Filter by both conditions = Bob and Cara [OK]

Hint: Check each condition separately, then combine results [OK]

Common Mistakes:

• Including Dave who has score 60
• Mixing up & and | operators
• Ignoring parentheses around conditions

4. What is wrong with this code snippet for filtering rows where score is above 80?

df[df['score'] > 80 & df['age'] < 30]

medium

A. Missing parentheses around conditions causing operator precedence error

B. Using & instead of | for combining conditions

C. Incorrect column name 'score'

D. DataFrame df is not defined

Solution

1. Step 1: Understand operator precedence in pandas boolean indexing

   Bitwise operators (&, |) require parentheses around each condition to avoid errors.

2. Step 2: Identify missing parentheses

   Code lacks parentheses around df['score'] > 80 and df['age'] < 30, causing wrong evaluation.

3. Final Answer:

   Missing parentheses around conditions causing operator precedence error -> Option A

4. Quick Check:

   Use parentheses with & and | [OK]

Hint: Always wrap each condition in parentheses when using & or | [OK]

Common Mistakes:

• Forgetting parentheses around each condition
• Confusing & with logical and (and)
• Assuming no error without parentheses

5. You have a DataFrame df with columns 'city', 'temperature', and 'humidity'. You want to select rows where city is either 'Paris' or 'London' and temperature is above 20. Which code correctly applies boolean indexing?

hard

A. df[((df['city'] == 'Paris') | (df['city'] == 'London')) & (df['temperature'] > 20)]

B. df[df['city'] == 'Paris' | df['city'] == 'London' & (df['temperature'] > 20)]

C. df[df['city'] == ('Paris' or 'London') & df['temperature'] > 20]

D. df[(df['city'] == 'Paris') | (df['city'] == 'London') & (df['temperature'] > 20)]

Solution

1. Step 1: Understand condition grouping with parentheses

   Conditions combined with | must be wrapped in parentheses to group city checks correctly.

2. Step 2: Check each option for correct syntax and logic

   df[((df['city'] == 'Paris') | (df['city'] == 'London')) & (df['temperature'] > 20)] correctly groups city conditions and combines with temperature condition using &.

3. Final Answer:

   df[((df['city'] == 'Paris') | (df['city'] == 'London')) & (df['temperature'] > 20)] -> Option A

4. Quick Check:

   Group OR conditions with parentheses before AND [OK]

Hint: Group OR conditions in parentheses before AND conditions [OK]

Common Mistakes:

• Missing parentheses causing wrong condition order
• Using Python 'or' instead of |
• Trying to compare city to ('Paris' or 'London') directly