filter() for group-level filtering in Data Analysis Python - Mini Project: Build & Apply

Filter Groups Using filter() in Python

📖 Scenario: You work in a small bookstore. You have a list of sales records. Each record shows the genre of the book sold and the number of copies sold that day.You want to find which genres sold more than 20 copies in total.

🎯 Goal: Build a Python program that groups sales by genre and uses filter() to keep only genres with total sales above 20.

📋 What You'll Learn

Create a list of sales records as tuples with genre and copies sold

Create a function to sum copies sold per genre

Use filter() to keep genres with total sales above a threshold

Print the filtered genres and their total sales

💡 Why This Matters

🌍 Real World

Filtering grouped data is common in sales analysis, customer segmentation, and many business reports.

💼 Career

Data analysts and scientists often need to filter groups of data based on summary statistics to find important trends.

1

Create the sales data list

Create a list called sales with these exact tuples: ("Fiction", 10), ("Science", 15), ("Fiction", 12), ("History", 8), ("Science", 10), ("History", 15).

Data Analysis Python

# Create the sales list with genre and copies sold tuples
# Your code here

Hint

Use a list of tuples. Each tuple has a genre string and an integer for copies sold.

2

Create a dictionary to sum sales by genre

Create a dictionary called genre_totals. Use a for loop to add copies sold for each genre from sales. Initialize missing genres with 0 before adding.

Data Analysis Python

sales = [("Fiction", 10), ("Science", 15), ("Fiction", 12), ("History", 8), ("Science", 10), ("History", 15)]
# Create genre_totals dictionary and sum copies sold per genre
# Your code here

Hint

Use a dictionary to keep totals. Check if genre is in dictionary before adding.

3

Use filter() to keep genres with total sales above 20

Create a function called above_20 that takes a genre_total tuple and returns True if the total sales (second item) is greater than 20. Then use filter() with above_20 on genre_totals.items() and convert the result to a list called filtered_genres.

Data Analysis Python

sales = [("Fiction", 10), ("Science", 15), ("Fiction", 12), ("History", 8), ("Science", 10), ("History", 15)]
genre_totals = {}
for genre, copies in sales:
    if genre not in genre_totals:
        genre_totals[genre] = 0
    genre_totals[genre] += copies

# Define above_20 function and use filter() to create filtered_genres list
# Your code here

Hint

The function gets a tuple like ("Fiction", 22). Check if the second value is greater than 20.

4

Print the filtered genres and their total sales

Write a print() statement to display filtered_genres.

Data Analysis Python

sales = [("Fiction", 10), ("Science", 15), ("Fiction", 12), ("History", 8), ("Science", 10), ("History", 15)]
genre_totals = {}
for genre, copies in sales:
    if genre not in genre_totals:
        genre_totals[genre] = 0
    genre_totals[genre] += copies

def above_20(genre_total):
    return genre_total[1] > 20

filtered_genres = list(filter(above_20, genre_totals.items()))

# Print the filtered_genres list
# Your code here

Hint

Just print the filtered_genres list to see the genres with sales above 20.

Practice

1. What does the filter() function do when used with groupby in pandas?

easy

A. Filters individual rows inside groups without removing groups

B. Keeps or removes entire groups based on a condition function

C. Sorts groups by their size

D. Aggregates data by summing values in each group

Solution

1. Step 1: Understand filter() with groupby

   The filter() function applies a condition to each group and keeps or removes the whole group based on True or False.

2. Step 2: Differentiate from other functions

   Unlike aggregation or sorting, filter() does not change group content but decides if the group stays or goes.

3. Final Answer:

   Keeps or removes entire groups based on a condition function -> Option B

4. Quick Check:

   filter() = group-level keep/remove [OK]

Hint: filter() keeps or drops whole groups by condition [OK]

Common Mistakes:

• Thinking filter() works on individual rows only
• Confusing filter() with aggregation functions
• Assuming filter() sorts groups

2. Which of the following is the correct syntax to filter groups in a pandas DataFrame df grouped by column 'Category' where the group size is greater than 3?

easy

A. df.groupby('Category').filter(lambda x: len(x) > 3)

B. df.filter(groupby='Category', condition=lambda x: len(x) > 3)

C. df.groupby('Category').apply(lambda x: len(x) > 3)

D. df.groupby('Category').select(lambda x: len(x) > 3)

Solution

1. Step 1: Recall filter() syntax with groupby

   The correct syntax is df.groupby('Category').filter(function) where the function returns True or False per group.

2. Step 2: Check each option

   df.groupby('Category').filter(lambda x: len(x) > 3) matches the correct syntax. Options A, C, and D use invalid methods or parameters.

3. Final Answer:

   df.groupby('Category').filter(lambda x: len(x) > 3) -> Option A

4. Quick Check:

   Correct syntax uses groupby().filter() [OK]

Hint: Use groupby().filter(lambda x: condition) syntax [OK]

Common Mistakes:

• Using filter() directly on DataFrame without groupby
• Using apply() instead of filter() for filtering groups
• Passing wrong parameters to filter()

3. Given the following code, what will be the output?

import pandas as pd
data = {'Team': ['A', 'A', 'B', 'B', 'B', 'C'], 'Points': [10, 15, 5, 7, 8, 20]}
df = pd.DataFrame(data)
filtered = df.groupby('Team').filter(lambda x: x['Points'].mean() > 10)
print(filtered)

medium

A. Rows where Team is 'A' or 'C' only

B. Rows where Team is 'B' only

C. All rows from all teams

D. Empty DataFrame

Solution

1. Step 1: Calculate mean Points per Team

   Team A mean = (10+15)/2 = 12.5, Team B mean = (5+7+8)/3 = 6.67, Team C mean = 20

2. Step 2: Apply filter condition

   Keep groups with mean Points > 10, so Teams A and C remain, Team B is removed.

3. Final Answer:

   Rows where Team is 'A' or 'C' only -> Option A

4. Quick Check:

   Mean > 10 keeps A and C [OK]

Hint: Calculate group mean, keep groups meeting condition [OK]

Common Mistakes:

• Forgetting to calculate mean per group
• Assuming filter keeps rows, not groups
• Confusing group means with total sums

4. Identify the error in this code snippet that tries to filter groups with sum of 'Sales' greater than 100:

df.groupby('Region').filter(lambda x: x['Sales'].sum > 100)

medium

A. Lambda function should return False instead of True

B. filter() cannot be used with groupby

C. Missing parentheses after sum method call

D. Wrong column name 'Sales'

Solution

1. Step 1: Check the lambda function syntax

   The code uses x['Sales'].sum which is a method, but it lacks parentheses to call it.

2. Step 2: Correct the method call

   It should be x['Sales'].sum() to get the sum value for comparison.

3. Final Answer:

   Missing parentheses after sum method call -> Option C

4. Quick Check:

   sum() needs parentheses to execute [OK]

Hint: Remember to call sum() with parentheses [OK]

Common Mistakes:

• Using sum without parentheses
• Thinking filter() is invalid with groupby
• Assuming wrong column name causes error

5. You have a sales DataFrame df with columns 'Store', 'Month', and 'Revenue'. You want to keep only stores where the total revenue in any month exceeds 5000. Which code correctly filters these stores using filter()?

hard

A. df.groupby('Store').filter(lambda g: g['Revenue'].mean() > 5000)

B. df.groupby('Store').filter(lambda g: g['Revenue'].sum() > 5000)

C. df.groupby('Month').filter(lambda g: g['Revenue'].max() > 5000)

D. df.groupby('Store').filter(lambda g: g['Revenue'].max() > 5000)

Solution

1. Step 1: Understand the condition

   We want stores where revenue in any month exceeds 5000, so check max revenue per store.

2. Step 2: Analyze each option

   df.groupby('Store').filter(lambda g: g['Revenue'].max() > 5000) uses max() per store, matching the condition. df.groupby('Store').filter(lambda g: g['Revenue'].sum() > 5000) sums all revenue, which is different. df.groupby('Month').filter(lambda g: g['Revenue'].max() > 5000) groups by Month, not Store. df.groupby('Store').filter(lambda g: g['Revenue'].mean() > 5000) uses mean, which is not the condition.

3. Final Answer:

   df.groupby('Store').filter(lambda g: g['Revenue'].max() > 5000) -> Option D

4. Quick Check:

   max() per store matches condition [OK]

Hint: Use max() to check if any month exceeds threshold [OK]

Common Mistakes:

• Using sum() instead of max() for condition
• Grouping by wrong column
• Using mean() which may not meet condition