filter() for group-level filtering in Data Analysis Python - Step-by-Step Execution

Concept Flow - filter() for group-level filtering

Start with DataFrame

↓

Group data by key

↓

Apply filter function to each group

↓

Keep groups where filter returns True

↓

Combine filtered groups into new DataFrame

↓

Result: Filtered grouped data

We start with data, group it by a key, then apply a filter function to each group. Only groups passing the filter stay in the result.

Execution Sample

Data Analysis Python

import pandas as pd

df = pd.DataFrame({
    'Team': ['A', 'A', 'B', 'B', 'C'],
    'Points': [10, 15, 5, 7, 20]
})

filtered = df.groupby('Team').filter(lambda g: g['Points'].sum() > 20)
print(filtered)

This code groups data by 'Team' and keeps only teams with total Points over 20.

Execution Table

Step	Group	Group Points	Sum Points	Filter Condition (sum > 20)	Keep Group?
1	A	[10, 15]	25	25 > 20 is True	Yes
2	B	[5, 7]	12	12 > 20 is False	No
3	C	[20]	20	20 > 20 is False	No
4	Combine kept groups	-	-	-	Only group A kept

💡 All groups checked; only group A meets condition and is kept.

Variable Tracker

Variable	Start	After Group A	After Group B	After Group C	Final
filtered	empty	contains rows of Team A	no change	no change	DataFrame with Team A rows only

Key Moments - 2 Insights

Why does group C get removed even though it has 20 points?

Because the filter condition is sum > 20, not >= 20. Group C's sum is exactly 20, so it fails the condition (see execution_table row 3).

Does filter() keep individual rows or whole groups?

filter() keeps or removes entire groups based on the condition applied to the group as a whole, not individual rows (see execution_table rows 1-3).

Visual Quiz - 3 Questions

Look at the execution table, what is the sum of points for group B at step 2?

A 7

B 12

C 5

D 20

💡 Hint

Check the 'Sum Points' column in execution_table row 2.

At which step does the filter condition evaluate to False for group C?

A Step 1

B Step 2

C Step 3

D Step 4

💡 Hint

Look at the 'Filter Condition' column for group C in execution_table.

If the filter condition changed to sum >= 20, which groups would be kept?

A Groups A and C

B Groups B and C

C Only group A

D All groups

💡 Hint

Compare sum points and condition in execution_table rows 1 and 3.

Concept Snapshot

filter() for group-level filtering:
- Use df.groupby('key').filter(func)
- func gets each group DataFrame
- Return True to keep group, False to drop
- Keeps whole groups, not rows
- Useful to filter groups by aggregate stats

Full Transcript

We start with a DataFrame and group it by a key column. Then, we apply a filter function to each group. This function checks a condition on the group, like if the sum of a column is greater than a value. Groups passing the condition are kept whole; others are removed. For example, grouping by 'Team' and filtering teams with total points over 20 keeps only those teams. The filter function returns True or False per group, not per row. This helps us keep or remove entire groups based on group-level statistics.