Iterating over groups in Pandas - Time & Space Complexity

Time Complexity: Iterating over groups

O(n)

Understanding Time Complexity

When we split data into groups and look at each group one by one, it takes time. We want to know how this time grows as the data gets bigger.

How does the time to go through all groups change when we have more data?

Scenario Under Consideration

Analyze the time complexity of the following code snippet.

import pandas as pd

df = pd.DataFrame({
    'Category': ['A', 'B', 'A', 'B', 'C'],
    'Value': [10, 20, 30, 40, 50]
})

groups = df.groupby('Category')

for name, group in groups:
    print(name, group['Value'].sum())

This code splits the data by 'Category' and then goes through each group to add up the 'Value' numbers.

Identify Repeating Operations

Identify the loops, recursion, array traversals that repeat.

• Primary operation: Looping over each group created by the split.
• How many times: Once for each unique category in the data.

How Execution Grows With Input

As the data grows, the number of groups and the size of each group can change. The code visits every row once when grouping and then once more when summing values in each group.

Input Size (n)	Approx. Operations
10	About 10 operations to group and sum
100	About 100 operations to group and sum
1000	About 1000 operations to group and sum

Pattern observation: The total work grows roughly in direct proportion to the number of rows.

Final Time Complexity

Time Complexity: O(n)

This means the time to process grows linearly with the number of rows in the data.

Common Mistake

[X] Wrong: "Grouping data is instant and does not add to the time cost."

[OK] Correct: Grouping requires looking at every row to decide which group it belongs to, so it takes time proportional to the data size.

Interview Connect

Understanding how grouping and iterating over groups affects time helps you explain your code choices clearly and shows you know how data size impacts performance.

Self-Check

"What if we replaced the loop over groups with a vectorized aggregation? How would the time complexity change?"

Practice

1. What does the groupby method in pandas do when used on a DataFrame?

easy

A. It changes the data types of columns.

B. It sorts the DataFrame by a column.

C. It removes duplicate rows from the DataFrame.

D. It splits the DataFrame into groups based on column values.

Solution

1. Step 1: Understand the purpose of groupby

   The groupby method is used to split data into groups based on values in one or more columns.

2. Step 2: Compare with other options

   Sorting, removing duplicates, and changing data types are different operations, not what groupby does.

3. Final Answer:

   It splits the DataFrame into groups based on column values. -> Option D

4. Quick Check:

   groupby = split into groups [OK]

Hint: Remember: groupby splits data by column values [OK]

Common Mistakes:

• Confusing groupby with sorting
• Thinking groupby removes duplicates
• Assuming groupby changes data types

2. Which of the following is the correct syntax to iterate over groups in a pandas DataFrame df grouped by column 'Category'?

easy

A. for name, group in df.groupby('Category'):

B. for group, name in df.groupby('Category'):

C. df.groupby('Category').for name, group:

D. for name in df.groupby('Category'):

Solution

1. Step 1: Recall groupby iteration syntax

   When iterating, the loop returns a tuple: (group name, group DataFrame).

2. Step 2: Match syntax to correct order

   for name, group in df.groupby('Category'): correctly uses for name, group in df.groupby('Category'):. Others have wrong order or syntax.

3. Final Answer:

   for name, group in df.groupby('Category'): -> Option A

4. Quick Check:

   for name, group in groupby [OK]

Hint: Use 'for name, group in df.groupby()' to iterate groups [OK]

Common Mistakes:

• Swapping group and name variables
• Using invalid for loop syntax
• Ignoring that groupby returns tuples

3. Given the DataFrame df below:

  Category  Value
0        A      1
1        B      2
2        A      3
3        B      4
4        C      5

What will be the output of this code?

for name, group in df.groupby('Category'):
    print(name, group['Value'].sum())

medium

A. A 4
B 2
C 5

B. A 4
B 6
C 5

C. A 3
B 6
C 5

D. A 1
B 4
C 5

Solution

1. Step 1: Group data by 'Category'

   Groups are: A with values [1,3], B with [2,4], C with [5].

2. Step 2: Sum 'Value' in each group

   Sum for A = 1+3=4, B = 2+4=6, C = 5.

3. Final Answer:

   A 4 B 6 C 5 -> Option B

4. Quick Check:

   Sum values per group = output [OK]

Hint: Sum values inside each group to find output [OK]

Common Mistakes:

• Summing only first value per group
• Mixing group names and sums
• Ignoring groupby splits

4. What is wrong with this code snippet that tries to iterate over groups?

for group in df.groupby('Type'):
    print(group['Value'].mean())

medium

A. It should unpack the tuple into name and group before accessing 'Value'.

B. The groupby method requires a list, not a string.

C. The print statement should be outside the loop.

D. The mean() function cannot be used on groups.

Solution

1. Step 1: Understand groupby iteration output

   Each iteration returns a tuple: (group name, group DataFrame).

2. Step 2: Identify error in code

   The code uses only one variable 'group', so 'group' is a tuple, not a DataFrame. Accessing group['Value'] causes error.

3. Final Answer:

   It should unpack the tuple into name and group before accessing 'Value'. -> Option A

4. Quick Check:

   Unpack tuple as (name, group) [OK]

Hint: Unpack groupby output as (name, group) to avoid errors [OK]

Common Mistakes:

• Using one variable instead of two in for loop
• Trying to access DataFrame methods on tuple
• Misunderstanding groupby output structure

5. You have a DataFrame df with columns 'Team' and 'Score'. You want to create a dictionary where keys are team names and values are the average score of each team. Which code correctly does this using iteration over groups?

hard

A. result = {team: group['Score'].mean() for team, group in df.iterrows()}

B. result = {}
for group in df.groupby('Team'):
result[group] = group['Score'].mean()

C. result = {}
for team, group in df.groupby('Team'):
result[team] = group['Score'].mean()

D. result = df.groupby('Team')['Score'].mean().to_dict()

Solution

1. Step 1: Understand iteration over groups

   Using for team, group in df.groupby('Team') correctly unpacks group name and data.

2. Step 2: Calculate mean and store in dictionary

   Inside loop, compute mean of 'Score' and assign to dictionary with team as key.

3. Final Answer:

   result = {} for team, group in df.groupby('Team'): result[team] = group['Score'].mean() -> Option C

4. Quick Check:

   Iterate groups, compute mean, store in dict [OK]

Hint: Use for team, group in groupby and assign mean to dict [OK]

Common Mistakes:

• Not unpacking groupby tuple
• Using iterrows instead of groupby
• Trying to assign group tuple as dict key