GroupBy performance considerations in Pandas - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of GroupBy with multiple aggregations

What is the output of this code snippet using pandas GroupBy with multiple aggregations?

Pandas

import pandas as pd

df = pd.DataFrame({
    'Category': ['A', 'A', 'B', 'B', 'C'],
    'Value': [10, 20, 10, 30, 40]
})
result = df.groupby('Category').agg({'Value': ['sum', 'mean']})
print(result)

A

Category  Value
sum       30
mean      15.0

B

          Value     
           sum  mean
Category           
A           30  15.0
B           40  20.0
C           40  40.0

C

Value
sum    100
mean    20.0

D

Category  Value
A        30
B        40
C        40

💡 Hint

Remember that multiple aggregations create a MultiIndex column in the result.

When using multiple aggregation functions on a column, pandas returns a DataFrame with MultiIndex columns showing each aggregation result per group.

❓ data_output

intermediate

Number of groups created by GroupBy

Given this DataFrame, how many groups will pandas GroupBy create when grouping by 'Type'?

Pandas

import pandas as pd

df = pd.DataFrame({
    'Type': ['X', 'Y', 'X', 'Z', 'Y', 'X'],
    'Score': [5, 10, 15, 20, 25, 30]
})
groups = df.groupby('Type')
print(len(groups))

A 3

B 6

C 4

D 1

💡 Hint

Count unique values in the 'Type' column.

GroupBy creates one group per unique value in the grouping column. Here, 'X', 'Y', and 'Z' are unique, so 3 groups.

🔧 Debug

advanced

Identify the cause of slow GroupBy operation

This code runs very slowly on a large DataFrame. What is the main reason for the slow performance?

Pandas

import pandas as pd
import numpy as np

df = pd.DataFrame({
    'Category': np.random.choice(['A', 'B', 'C', 'D'], size=10_000_000),
    'Value': np.random.rand(10_000_000)
})

result = df.groupby('Category').apply(lambda x: x['Value'].sum())

A Using apply with a lambda function instead of built-in aggregation slows down performance.

B Grouping by a categorical column always causes slowdowns.

C The DataFrame is too small to benefit from GroupBy optimizations.

D The random values in 'Value' column cause the slowdown.

💡 Hint

Built-in aggregation functions are faster than apply with custom functions.

Using apply with a lambda function prevents pandas from using optimized C code for aggregation, causing slow performance on large data.

🧠 Conceptual

advanced

Effect of sorting on GroupBy performance

How does setting the 'sort' parameter to False in pandas GroupBy affect performance and output?

A Setting sort=False removes duplicate groups, improving performance.

B Setting sort=False causes an error because sorting is mandatory.

C Setting sort=False sorts groups alphabetically, slowing down performance.

D Setting sort=False improves performance by skipping sorting but keeps groups in original order.

💡 Hint

Sorting groups is optional and can be skipped for speed.

By default, GroupBy sorts group keys which adds overhead. Setting sort=False skips sorting, speeding up grouping and preserving original order.

🚀 Application

expert

Optimizing memory usage in GroupBy with large categorical data

You have a DataFrame with 50 million rows and a 'Category' column with 100 unique values. Which approach best optimizes memory and performance for grouping?

A Convert 'Category' to string dtype before grouping.

B Keep 'Category' as object dtype and group directly.

C Convert 'Category' to pandas Categorical dtype before grouping.

D Drop the 'Category' column before grouping to save memory.

💡 Hint

Categorical dtype uses less memory and speeds up grouping on repeated values.

Converting to Categorical reduces memory by storing codes instead of strings and speeds up groupby operations on large data with repeated categories.

Practice

1. What is a key factor that affects the speed of pandas groupby operations?

easy

A. The type of operating system

B. The number of columns used to group the data

C. The font size of the code editor

D. The color of the data points

Solution

1. Step 1: Understand what affects groupby speed

   The more columns you group by, the more complex the operation becomes.

2. Step 2: Relate grouping columns to performance

   Grouping by fewer columns reduces computation and speeds up the process.

3. Final Answer:

   The number of columns used to group the data -> Option B

4. Quick Check:

   Grouping columns = performance impact [OK]

Hint: Fewer grouping columns usually mean faster groupby [OK]

Common Mistakes:

• Thinking system or editor settings affect groupby speed
• Ignoring the number of grouping columns
• Confusing data color or style with performance

2. Which of the following pandas groupby syntax examples is correct for grouping by column 'A' and calculating the mean of column 'B'?

easy

A. df.groupby('A')['B'].mean()

B. df.group_by('A').mean('B')

C. df.groupby('B')['A'].mean()

D. df.groupby('A').mean['B']

Solution

1. Step 1: Check correct groupby syntax

   The correct method is df.groupby('A')['B'].mean() to group by 'A' and get mean of 'B'.

2. Step 2: Identify syntax errors in other options

   Options A, B, and D have wrong method names or wrong column selections.

3. Final Answer:

   df.groupby('A')['B'].mean() -> Option A

4. Quick Check:

   Correct groupby syntax = df.groupby('A')['B'].mean() [OK]

Hint: Use df.groupby('col')['col2'].agg() format [OK]

Common Mistakes:

• Using group_by instead of groupby
• Swapping columns in groupby and aggregation
• Using wrong method call syntax

3. Given the code below, what will be the output?

import pandas as pd
df = pd.DataFrame({
  'Category': ['A', 'A', 'B', 'B', 'C'],
  'Value': [10, 20, 10, 30, 50]
})
result = df.groupby('Category').sum()
print(result)

medium

A. {'A': 30, 'B': 40, 'C': 50}

B. Category\nA 30\nB 40\nC 50

C. Value\nCategory\nA 30\nB 40\nC 50

D. Error due to missing aggregation

Solution

1. Step 1: Understand groupby sum output

   Grouping by 'Category' sums 'Value' for each group, resulting in sums 30, 40, and 50.

2. Step 2: Recognize pandas output format

   The output is a DataFrame with 'Value' as column and 'Category' as index, shown as:
   Value
   Category
   A 30
   B 40
   C 50

3. Final Answer:

   Value Category A 30 B 40 C 50 -> Option C

4. Quick Check:

   Group sum output = Value\nCategory\nA 30\nB 40\nC 50 [OK]

Hint: Groupby sum returns DataFrame with grouped index and summed columns [OK]

Common Mistakes:

• Expecting a dictionary instead of DataFrame
• Ignoring index and column labels in output
• Thinking sum needs explicit aggregation

4. What is wrong with this code snippet that groups by columns 'A' and 'B' and calculates the mean?

df.groupby(['A', 'B']).mean

medium

A. Mean cannot be used with multiple grouping columns

B. Wrong method name, should be avg()

C. Groupby should be called with a dictionary

D. Missing parentheses to call the mean function

Solution

1. Step 1: Check method call syntax

   The code uses mean without parentheses, so it returns a method object, not the result.

2. Step 2: Correct the syntax

   Adding parentheses mean() calls the function and returns the aggregated result.

3. Final Answer:

   Missing parentheses to call the mean function -> Option D

4. Quick Check:

   Function call needs () [OK]

Hint: Always add () to call aggregation functions [OK]

Common Mistakes:

• Forgetting parentheses after aggregation functions
• Using avg() instead of mean()
• Thinking groupby needs dict argument

5. You have a large DataFrame with columns 'City', 'Year', and 'Sales'. To improve groupby performance when calculating total sales per city, which approach is best?

hard

A. Group only by 'City' to reduce grouping complexity

B. Group by both 'City' and 'Year' to get detailed results

C. Sort the DataFrame by 'Sales' before grouping

D. Convert 'City' to string type before grouping

Solution

1. Step 1: Analyze grouping columns impact

   Grouping by fewer columns reduces the number of groups and speeds up computation.

2. Step 2: Choose grouping for total sales per city

   Since total sales per city is needed, grouping only by 'City' is sufficient and faster.

3. Final Answer:

   Group only by 'City' to reduce grouping complexity -> Option A

4. Quick Check:

   Less grouping columns = better performance [OK]

Hint: Group by only needed columns for faster results [OK]

Common Mistakes:

• Grouping by extra columns unnecessarily
• Sorting before grouping does not improve speed
• Changing data types without reason