GroupBy with custom functions in Pandas - Step-by-Step Execution

Concept Flow - GroupBy with custom functions

Start with DataFrame

↓

Group data by column(s)

↓

Apply custom function to each group

↓

Combine results into new DataFrame

↓

Output grouped summary

We start with a table, group rows by a column, apply a custom function to each group, then combine the results.

Execution Sample

Pandas

import pandas as pd

df = pd.DataFrame({'Team': ['A', 'A', 'B', 'B'], 'Points': [10, 15, 10, 20]})

def range_func(x):
    return x.max() - x.min()

result = df.groupby('Team')['Points'].apply(range_func)

This code groups points by team and applies a custom function to find the range of points in each team.

Execution Table

Step	Action	Group	Data in Group	Custom Function Applied	Result
1	Group data by 'Team'	A	[10, 15]	range_func([10, 15])	15 - 10 = 5
2	Group data by 'Team'	B	[10, 20]	range_func([10, 20])	20 - 10 = 10
3	Combine results	-	-	-	{'A': 5, 'B': 10}
4	End	-	-	-	Grouping and custom function complete

💡 All groups processed, results combined into final output

Variable Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	Final
df	{'Team': ['A','A','B','B'], 'Points': [10,15,10,20]}	Grouped by 'Team'	Grouped by 'Team'	Grouped by 'Team'	Unchanged
group 'A'	N/A	[10, 15]	[10, 15]	[10, 15]	Used for calculation
group 'B'	N/A	N/A	[10, 20]	[10, 20]	Used for calculation
result	N/A	N/A	N/A	{'A': 5, 'B': 10}	{'A': 5, 'B': 10}

Key Moments - 3 Insights

Why does the custom function receive only the 'Points' data for each group?

Because we select the 'Points' column before applying the function (df.groupby('Team')['Points'].apply), so the function gets only that column's data per group, as shown in execution_table rows 1 and 2.

How does pandas combine the results from each group after applying the custom function?

Pandas collects the output from each group's function call and creates a new Series indexed by the group keys, as seen in execution_table row 3.

What happens if the custom function returns a single value for each group?

Pandas creates a Series with group keys as index and the returned values as data, like the final 'result' variable in variable_tracker.

Visual Quiz - 3 Questions

Look at the execution_table at Step 2, what is the result of applying the custom function to group 'B'?

A 5

B 15

C 10

D 20

💡 Hint

Check the 'Result' column for Step 2 in the execution_table.

At which step does pandas combine the results from all groups?

A Step 3

B Step 1

C Step 2

D Step 4

💡 Hint

Look for the step where the 'Combine results' action happens in the execution_table.

If the custom function returned the sum instead of the range, what would be the result for group 'A'?

A 15

B 25

C 10

D 5

💡 Hint

Sum of [10, 15] is 25; check variable_tracker for group 'A' data.

Concept Snapshot

GroupBy with custom functions:
- Use df.groupby('col')['target'].apply(func)
- func gets each group's target column data
- func returns a value per group
- pandas combines results into Series
- Useful for custom summaries beyond built-in agg

Full Transcript

We start with a DataFrame and group rows by a column, here 'Team'. For each group, we select the 'Points' column and apply a custom function that calculates the range (max - min). The function runs separately on each group's points. The results are combined into a new Series indexed by team names. This process lets us summarize data in flexible ways using our own functions.

Practice

1. What does the groupby function in pandas primarily do?

easy

A. It splits data into groups based on column values for separate analysis.

B. It sorts the entire DataFrame by column values.

C. It deletes duplicate rows from the DataFrame.

D. It merges two DataFrames based on a key column.

Solution

1. Step 1: Understand the purpose of groupby

   The groupby function divides data into groups based on unique values in one or more columns.

2. Step 2: Recognize the use of groups

   After grouping, you can apply functions to each group separately for analysis or aggregation.

3. Final Answer:

   It splits data into groups based on column values for separate analysis. -> Option A

4. Quick Check:

   Group data = split by column values [OK]

Hint: GroupBy splits data into parts by column values [OK]

Common Mistakes:

• Confusing groupby with sorting
• Thinking groupby deletes duplicates
• Mixing groupby with merge operations

2. Which of the following is the correct syntax to apply a custom function my_func to each group after grouping by column 'Category' in a DataFrame df?

easy

A. df.groupby('Category').apply(my_func)

B. df.apply.groupby('Category')(my_func)

C. df.groupby('Category').my_func()

D. df.group_by('Category').apply(my_func)

Solution

1. Step 1: Recall pandas groupby syntax

   The correct method to group by a column is df.groupby('ColumnName').

2. Step 2: Apply a custom function to groups

   Use .apply(function) after grouping to run the function on each group.

3. Final Answer:

   df.groupby('Category').apply(my_func) -> Option A

4. Quick Check:

   groupby + apply(custom function) = df.groupby('Category').apply(my_func) [OK]

Hint: Use groupby('col').apply(func) to run custom function [OK]

Common Mistakes:

• Using wrong method name like group_by
• Calling function directly on groupby object
• Misplacing apply before groupby

3. Given the DataFrame df below:

import pandas as pd
data = {'Team': ['A', 'A', 'B', 'B', 'C'], 'Points': [10, 15, 10, 20, 30]}
df = pd.DataFrame(data)

# Custom function to get range of points in group
def point_range(x):
    return x.max() - x.min()

result = df.groupby('Team')['Points'].apply(point_range).to_dict()

What is the value of result?

medium

A. {'A': 10, 'B': 10, 'C': 0}

B. {'A': 25, 'B': 30, 'C': 30}

C. {'A': 5, 'B': 10, 'C': 0}

D. {'A': 15, 'B': 20, 'C': 30}

Solution

1. Step 1: Group points by 'Team'

   Teams: A has points [10, 15], B has [10, 20], C has [30].

2. Step 2: Calculate range (max - min) for each group

   For A: 15 - 10 = 5; B: 20 - 10 = 10; C: 30 - 30 = 0.

3. Final Answer:

   {'A': 5, 'B': 10, 'C': 0} -> Option C

4. Quick Check:

   Range = max - min per group = {'A': 5, 'B': 10, 'C': 0} [OK]

Hint: Calculate max-min per group for range [OK]

Common Mistakes:

• Confusing sum with range
• Forgetting single value group range is zero
• Mixing keys and values in result

4. Consider this code snippet:

import pandas as pd
data = {'Category': ['X', 'X', 'Y'], 'Value': [1, 2, 3]}
df = pd.DataFrame(data)

# Custom function to sum values
 def sum_values(x):
    return x.sum

result = df.groupby('Category')['Value'].apply(sum_values)

What is the issue with this code?

medium

A. The apply method cannot be used with custom functions.

B. The groupby syntax is incorrect; it should be groupby(['Category']).

C. The DataFrame is missing a required index column.

D. The function sum_values returns the method, not the sum result.

Solution

1. Step 1: Check the custom function definition

   The function returns x.sum which is a method, not called (missing parentheses).

2. Step 2: Understand the effect on apply

   Returning the method itself causes the result to be a method object, not the sum value.

3. Final Answer:

   The function sum_values returns the method, not the sum result. -> Option D

4. Quick Check:

   Missing () calls method, not result [OK]

Hint: Call sum() with parentheses to get result [OK]

Common Mistakes:

• Forgetting parentheses on sum()
• Thinking groupby syntax is wrong here
• Believing apply can't use custom functions

5. You have a DataFrame df with columns 'Store', 'Product', and 'Sales'. You want to find the difference between the highest and lowest sales per store, but only for products with sales above 100. Which approach correctly uses groupby with a custom function to achieve this?

hard

A. df.groupby('Store')['Sales'].apply(lambda x: x[x > 100].max() - x[x > 100].min())

B. All of the above produce the same correct result.

C. df.groupby('Store').apply(lambda g: g[g['Sales'] > 100]['Sales'].max() - g[g['Sales'] > 100]['Sales'].min())

D. df[df['Sales'] > 100].groupby('Store')['Sales'].apply(lambda x: x.max() - x.min())

Solution

1. Step 1: Understand filtering sales above 100

   One approach filters inside the apply lambda on each store's sales series. Another filters the DataFrame before grouping. The third filters inside the group apply on the subgroup DataFrame.

2. Step 2: Check if all approaches calculate max-min correctly

   All three methods correctly compute the difference of max and min sales above 100 per store.

3. Step 3: Confirm equivalence of results

   Filtering before or inside groupby leads to the same final result for this case.

4. Final Answer:

   All of the above produce the same correct result. -> Option B

5. Quick Check:

   Filtering + groupby + custom function = all correct [OK]

Hint: Filter before or inside groupby; both work [OK]

Common Mistakes:

• Filtering after groupby without apply
• Not filtering sales > 100 properly
• Assuming only one method works