filter() for group-level filtering in Data Analysis Python - Cheat Sheet & Quick Revision

Recall & Review

beginner

What does the filter() function do when used with groups in data analysis?

It keeps or removes entire groups based on a condition applied to the group as a whole, not individual rows.

beginner

How do you apply filter() to keep groups where the group size is greater than 3?

Use df.groupby('column').filter(lambda x: len(x) > 3) to keep groups with more than 3 rows.

beginner

True or False: filter() can be used to keep groups based on the average value of a column in that group.

True. You can write a condition like lambda x: x['col'].mean() > value inside filter().

intermediate

What type of object does filter() receive when used after groupby()?

It receives a DataFrame representing each group, allowing you to check group-level properties.

intermediate

Why is filter() useful compared to filtering rows individually after grouping?

Because it lets you keep or remove whole groups based on group-level criteria, making group analysis easier.

What does df.groupby('A').filter(lambda x: len(x) > 2) do?

A Removes groups with more than 2 rows

B Keeps groups where the number of rows is more than 2

C Keeps rows where column 'A' is greater than 2

D Filters rows where length of column 'A' is greater than 2

The filter keeps entire groups where the group size (number of rows) is greater than 2.

Which of these is a valid condition inside filter() for group-level filtering?

A lambda x: x['score'].mean() > 50

B lambda x: x['score'] > 50

C lambda x: x['score'].sum()

D lambda x: x['score'].max()

The condition must return True or False. Only option A returns a boolean condition.

What type of object is passed to the function inside filter() after groupby()?

A A DataFrame for each group

B A Series of group keys

C A list of group names

D A single row from the DataFrame

The function receives each group's DataFrame to check group-level conditions.

If you want to keep groups where the maximum value in column 'sales' is above 100, which filter condition is correct?

A lambda x: x['sales'].mean() > 100

B lambda x: x['sales'] > 100

C lambda x: x['sales'].max() > 100

D lambda x: len(x) > 100

Option C checks the max value in 'sales' per group and returns True if above 100.

What happens to groups that do not meet the filter() condition?

A They are sorted to the end

B Only some rows are removed from the group

C They are marked but kept

D They are removed entirely from the result

Groups failing the condition are completely removed from the filtered DataFrame.

Explain how filter() works with groupby() to keep or remove groups.

Think about how you decide which friend groups to keep based on a group trait.

Describe a real-life example where group-level filtering with filter() would be useful.

Imagine sorting groups of items or people based on a group property.

Practice

1. What does the filter() function do when used with groupby in pandas?

easy

A. Filters individual rows inside groups without removing groups

B. Keeps or removes entire groups based on a condition function

C. Sorts groups by their size

D. Aggregates data by summing values in each group

Solution

1. Step 1: Understand filter() with groupby

   The filter() function applies a condition to each group and keeps or removes the whole group based on True or False.

2. Step 2: Differentiate from other functions

   Unlike aggregation or sorting, filter() does not change group content but decides if the group stays or goes.

3. Final Answer:

   Keeps or removes entire groups based on a condition function -> Option B

4. Quick Check:

   filter() = group-level keep/remove [OK]

Hint: filter() keeps or drops whole groups by condition [OK]

Common Mistakes:

• Thinking filter() works on individual rows only
• Confusing filter() with aggregation functions
• Assuming filter() sorts groups

2. Which of the following is the correct syntax to filter groups in a pandas DataFrame df grouped by column 'Category' where the group size is greater than 3?

easy

A. df.groupby('Category').filter(lambda x: len(x) > 3)

B. df.filter(groupby='Category', condition=lambda x: len(x) > 3)

C. df.groupby('Category').apply(lambda x: len(x) > 3)

D. df.groupby('Category').select(lambda x: len(x) > 3)

Solution

1. Step 1: Recall filter() syntax with groupby

   The correct syntax is df.groupby('Category').filter(function) where the function returns True or False per group.

2. Step 2: Check each option

   df.groupby('Category').filter(lambda x: len(x) > 3) matches the correct syntax. Options A, C, and D use invalid methods or parameters.

3. Final Answer:

   df.groupby('Category').filter(lambda x: len(x) > 3) -> Option A

4. Quick Check:

   Correct syntax uses groupby().filter() [OK]

Hint: Use groupby().filter(lambda x: condition) syntax [OK]

Common Mistakes:

• Using filter() directly on DataFrame without groupby
• Using apply() instead of filter() for filtering groups
• Passing wrong parameters to filter()

3. Given the following code, what will be the output?

import pandas as pd
data = {'Team': ['A', 'A', 'B', 'B', 'B', 'C'], 'Points': [10, 15, 5, 7, 8, 20]}
df = pd.DataFrame(data)
filtered = df.groupby('Team').filter(lambda x: x['Points'].mean() > 10)
print(filtered)

medium

A. Rows where Team is 'A' or 'C' only

B. Rows where Team is 'B' only

C. All rows from all teams

D. Empty DataFrame

Solution

1. Step 1: Calculate mean Points per Team

   Team A mean = (10+15)/2 = 12.5, Team B mean = (5+7+8)/3 = 6.67, Team C mean = 20

2. Step 2: Apply filter condition

   Keep groups with mean Points > 10, so Teams A and C remain, Team B is removed.

3. Final Answer:

   Rows where Team is 'A' or 'C' only -> Option A

4. Quick Check:

   Mean > 10 keeps A and C [OK]

Hint: Calculate group mean, keep groups meeting condition [OK]

Common Mistakes:

• Forgetting to calculate mean per group
• Assuming filter keeps rows, not groups
• Confusing group means with total sums

4. Identify the error in this code snippet that tries to filter groups with sum of 'Sales' greater than 100:

df.groupby('Region').filter(lambda x: x['Sales'].sum > 100)

medium

A. Lambda function should return False instead of True

B. filter() cannot be used with groupby

C. Missing parentheses after sum method call

D. Wrong column name 'Sales'

Solution

1. Step 1: Check the lambda function syntax

   The code uses x['Sales'].sum which is a method, but it lacks parentheses to call it.

2. Step 2: Correct the method call

   It should be x['Sales'].sum() to get the sum value for comparison.

3. Final Answer:

   Missing parentheses after sum method call -> Option C

4. Quick Check:

   sum() needs parentheses to execute [OK]

Hint: Remember to call sum() with parentheses [OK]

Common Mistakes:

• Using sum without parentheses
• Thinking filter() is invalid with groupby
• Assuming wrong column name causes error

5. You have a sales DataFrame df with columns 'Store', 'Month', and 'Revenue'. You want to keep only stores where the total revenue in any month exceeds 5000. Which code correctly filters these stores using filter()?

hard

A. df.groupby('Store').filter(lambda g: g['Revenue'].mean() > 5000)

B. df.groupby('Store').filter(lambda g: g['Revenue'].sum() > 5000)

C. df.groupby('Month').filter(lambda g: g['Revenue'].max() > 5000)

D. df.groupby('Store').filter(lambda g: g['Revenue'].max() > 5000)

Solution

1. Step 1: Understand the condition

   We want stores where revenue in any month exceeds 5000, so check max revenue per store.

2. Step 2: Analyze each option

   df.groupby('Store').filter(lambda g: g['Revenue'].max() > 5000) uses max() per store, matching the condition. df.groupby('Store').filter(lambda g: g['Revenue'].sum() > 5000) sums all revenue, which is different. df.groupby('Month').filter(lambda g: g['Revenue'].max() > 5000) groups by Month, not Store. df.groupby('Store').filter(lambda g: g['Revenue'].mean() > 5000) uses mean, which is not the condition.

3. Final Answer:

   df.groupby('Store').filter(lambda g: g['Revenue'].max() > 5000) -> Option D

4. Quick Check:

   max() per store matches condition [OK]

Hint: Use max() to check if any month exceeds threshold [OK]

Common Mistakes:

• Using sum() instead of max() for condition
• Grouping by wrong column
• Using mean() which may not meet condition