Iterating over groups in Pandas

Introduction

We use iterating over groups to look at parts of data separately. It helps us understand or work with each group one by one.

You want to analyze sales data for each store separately.

You need to calculate average scores for each class in a school dataset.

You want to process customer data grouped by region.

You want to apply different operations to each group in your data.

Syntax

Pandas

for group_name, group_data in df.groupby('column_name'):
    # do something with group_name and group_data

group_name is the value of the group (like a category).

group_data is a smaller DataFrame with only rows from that group.

Examples

This prints each city name and the rows for that city.

Pandas

for city, data in df.groupby('City'):
    print(city)
    print(data)

This calculates and prints the average price for each category.

Pandas

for category, group in df.groupby('Category'):
    avg_price = group['Price'].mean()
    print(f"Average price in {category}: {avg_price}")

Sample Program

This program groups sales data by city and sums sales for each city. Then it prints the total sales per city.

Pandas

import pandas as pd

data = {'City': ['Paris', 'Paris', 'London', 'London', 'Berlin'],
        'Sales': [100, 150, 200, 250, 300]}
df = pd.DataFrame(data)

for city, group in df.groupby('City'):
    total_sales = group['Sales'].sum()
    print(f"Total sales in {city}: {total_sales}")

Important Notes

Group order depends on the sorting of the group keys.

You can group by multiple columns by passing a list, like df.groupby(['City', 'Category']).

Summary

Use groupby to split data into groups.

Iterate over groups with a for loop to work on each group separately.

Each group gives you the group name and the data rows for that group.

Practice

1. What does the groupby method in pandas do when used on a DataFrame?

easy

A. It changes the data types of columns.

B. It sorts the DataFrame by a column.

C. It removes duplicate rows from the DataFrame.

D. It splits the DataFrame into groups based on column values.

Solution

1. Step 1: Understand the purpose of groupby

   The groupby method is used to split data into groups based on values in one or more columns.

2. Step 2: Compare with other options

   Sorting, removing duplicates, and changing data types are different operations, not what groupby does.

3. Final Answer:

   It splits the DataFrame into groups based on column values. -> Option D

4. Quick Check:

   groupby = split into groups [OK]

Hint: Remember: groupby splits data by column values [OK]

Common Mistakes:

• Confusing groupby with sorting
• Thinking groupby removes duplicates
• Assuming groupby changes data types

2. Which of the following is the correct syntax to iterate over groups in a pandas DataFrame df grouped by column 'Category'?

easy

A. for name, group in df.groupby('Category'):

B. for group, name in df.groupby('Category'):

C. df.groupby('Category').for name, group:

D. for name in df.groupby('Category'):

Solution

1. Step 1: Recall groupby iteration syntax

   When iterating, the loop returns a tuple: (group name, group DataFrame).

2. Step 2: Match syntax to correct order

   for name, group in df.groupby('Category'): correctly uses for name, group in df.groupby('Category'):. Others have wrong order or syntax.

3. Final Answer:

   for name, group in df.groupby('Category'): -> Option A

4. Quick Check:

   for name, group in groupby [OK]

Hint: Use 'for name, group in df.groupby()' to iterate groups [OK]

Common Mistakes:

• Swapping group and name variables
• Using invalid for loop syntax
• Ignoring that groupby returns tuples

3. Given the DataFrame df below:

  Category  Value
0        A      1
1        B      2
2        A      3
3        B      4
4        C      5

What will be the output of this code?

for name, group in df.groupby('Category'):
    print(name, group['Value'].sum())

medium

A. A 4
B 2
C 5

B. A 4
B 6
C 5

C. A 3
B 6
C 5

D. A 1
B 4
C 5

Solution

1. Step 1: Group data by 'Category'

   Groups are: A with values [1,3], B with [2,4], C with [5].

2. Step 2: Sum 'Value' in each group

   Sum for A = 1+3=4, B = 2+4=6, C = 5.

3. Final Answer:

   A 4 B 6 C 5 -> Option B

4. Quick Check:

   Sum values per group = output [OK]

Hint: Sum values inside each group to find output [OK]

Common Mistakes:

• Summing only first value per group
• Mixing group names and sums
• Ignoring groupby splits

4. What is wrong with this code snippet that tries to iterate over groups?

for group in df.groupby('Type'):
    print(group['Value'].mean())

medium

A. It should unpack the tuple into name and group before accessing 'Value'.

B. The groupby method requires a list, not a string.

C. The print statement should be outside the loop.

D. The mean() function cannot be used on groups.

Solution

1. Step 1: Understand groupby iteration output

   Each iteration returns a tuple: (group name, group DataFrame).

2. Step 2: Identify error in code

   The code uses only one variable 'group', so 'group' is a tuple, not a DataFrame. Accessing group['Value'] causes error.

3. Final Answer:

   It should unpack the tuple into name and group before accessing 'Value'. -> Option A

4. Quick Check:

   Unpack tuple as (name, group) [OK]

Hint: Unpack groupby output as (name, group) to avoid errors [OK]

Common Mistakes:

• Using one variable instead of two in for loop
• Trying to access DataFrame methods on tuple
• Misunderstanding groupby output structure

5. You have a DataFrame df with columns 'Team' and 'Score'. You want to create a dictionary where keys are team names and values are the average score of each team. Which code correctly does this using iteration over groups?

hard

A. result = {team: group['Score'].mean() for team, group in df.iterrows()}

B. result = {}
for group in df.groupby('Team'):
result[group] = group['Score'].mean()

C. result = {}
for team, group in df.groupby('Team'):
result[team] = group['Score'].mean()

D. result = df.groupby('Team')['Score'].mean().to_dict()

Solution

1. Step 1: Understand iteration over groups

   Using for team, group in df.groupby('Team') correctly unpacks group name and data.

2. Step 2: Calculate mean and store in dictionary

   Inside loop, compute mean of 'Score' and assign to dictionary with team as key.

3. Final Answer:

   result = {} for team, group in df.groupby('Team'): result[team] = group['Score'].mean() -> Option C

4. Quick Check:

   Iterate groups, compute mean, store in dict [OK]

Hint: Use for team, group in groupby and assign mean to dict [OK]

Common Mistakes:

• Not unpacking groupby tuple
• Using iterrows instead of groupby
• Trying to assign group tuple as dict key