Pivot tables with pivot_table() in Data Analysis Python - Time & Space Complexity

Time Complexity: Pivot tables with pivot_table()

O(n)

Understanding Time Complexity

We want to understand how the time it takes to create a pivot table changes as the data grows.

Specifically, how does the pivot_table() function handle bigger data sets?

Scenario Under Consideration

Analyze the time complexity of the following code snippet.

import pandas as pd

data = pd.DataFrame({
    'Category': ['A', 'B', 'A', 'B', 'C'],
    'Value': [10, 20, 30, 40, 50]
})

pivot = data.pivot_table(index='Category', values='Value', aggfunc='sum')

This code creates a pivot table that sums values for each category.

Identify Repeating Operations

Identify the loops, recursion, array traversals that repeat.

• Primary operation: Scanning all rows in the data to group by category.
• How many times: Once for each row in the data (n times).

How Execution Grows With Input

As the number of rows grows, the time to group and sum grows roughly in direct proportion.

Input Size (n)	Approx. Operations
10	About 10 operations to group and sum
100	About 100 operations
1000	About 1000 operations

Pattern observation: Doubling the data roughly doubles the work done.

Final Time Complexity

Time Complexity: O(n)

This means the time grows linearly with the number of rows in the data.

Common Mistake

[X] Wrong: "Pivot tables take constant time no matter how big the data is."

[OK] Correct: The function must look at each row to group and sum, so more data means more work.

Interview Connect

Understanding how data grouping scales helps you explain your approach clearly when working with real data.

Self-Check

"What if we added multiple grouping columns? How would the time complexity change?"

Practice

1. What does the index parameter do in the pivot_table() function?

easy

A. It sets the rows of the pivot table.

B. It sets the columns of the pivot table.

C. It defines the aggregation function.

D. It filters the data before pivoting.

Solution

1. Step 1: Understand the role of index in pivot_table()

   The index parameter specifies which column(s) to use as the rows in the resulting pivot table.

2. Step 2: Differentiate from other parameters

   The columns parameter sets the columns, aggfunc sets the aggregation, and filtering is done separately.

3. Final Answer:

   It sets the rows of the pivot table. -> Option A

4. Quick Check:

   index = rows [OK]

Hint: Remember: index = rows, columns = columns [OK]

Common Mistakes:

• Confusing index with columns
• Thinking aggfunc sets rows
• Assuming index filters data

2. Which of the following is the correct syntax to create a pivot table with pandas using pivot_table()?

easy

A. pd.pivot_table(data, index='Category', columns='Year', aggfunc='sum')

B. pd.pivot(data, index='Category', columns='Year', aggfunc='sum')

C. pd.pivot_table(data, rows='Category', cols='Year', aggfunc='sum')

D. pd.pivot_table(data, groupby='Category', by='Year', aggfunc='sum')

Solution

1. Step 1: Recall the correct function and parameters

   The correct function is pd.pivot_table() with parameters index for rows and columns for columns.

2. Step 2: Check each option

   pd.pivot_table(data, index='Category', columns='Year', aggfunc='sum') uses correct function and parameters. Options B, C, and D use incorrect function names or parameter names.

3. Final Answer:

   pd.pivot_table(data, index='Category', columns='Year', aggfunc='sum') -> Option A

4. Quick Check:

   Correct function and parameters = pd.pivot_table(data, index='Category', columns='Year', aggfunc='sum') [OK]

Hint: Use index= for rows, columns= for columns in pivot_table [OK]

Common Mistakes:

• Using pd.pivot instead of pd.pivot_table
• Wrong parameter names like rows or cols
• Mixing groupby parameters with pivot_table

3. Given the DataFrame df below:

   Product  Region  Sales
0  A        East    100
1  B        East    150
2  A        West    200
3  B        West    250
4  A        East    50
5  B        West    100

What is the output of:

pd.pivot_table(df, index='Product', columns='Region', values='Sales', aggfunc='sum')

medium

A. Region East West
Product
A 150 300
B 250 350

B. Region East West
Product
A 150 200
B 150 350

C. Region East West
Product
A 150 200
B 250 350

D. Region East West
Product
A 150 200
B 150 250

Solution

1. Step 1: Group data by Product and Region, summing Sales

   For Product A and East: 100 + 50 = 150; Product A and West: 200; Product B and East: 150; Product B and West: 250 + 100 = 350.

2. Step 2: Arrange results in pivot table format

   Rows are Products, columns are Regions, values are summed Sales as calculated.

3. Final Answer:

   Region East West Product A 150 200 B 250 350 -> Option C

4. Quick Check:

   Sum Sales by Product and Region = Region East West Product A 150 200 B 250 350 [OK]

Hint: Sum values grouped by index and columns [OK]

Common Mistakes:

• Adding wrong sales values
• Mixing up East and West sums
• Using count instead of sum

4. What is wrong with this code snippet?

pd.pivot_table(df, index='Product', columns='Region', values='Sales', aggfunc='average')

medium

A. The index parameter must be a list, not a string.

B. The aggfunc 'average' is not a valid aggregation function.

C. The values parameter cannot be a column name.

D. The columns parameter should be 'cols' instead of 'columns'.

Solution

1. Step 1: Check the aggfunc parameter value

   The valid aggregation functions include 'mean', 'sum', 'count', etc. 'average' is not recognized by pandas.

2. Step 2: Verify other parameters

   columns is correct, values can be a column name string, and index can be a string or list.

3. Final Answer:

   The aggfunc 'average' is not a valid aggregation function. -> Option B

4. Quick Check:

   aggfunc must be valid like 'mean' [OK]

Hint: Use 'mean' not 'average' for aggregation [OK]

Common Mistakes:

• Using 'average' instead of 'mean'
• Changing columns to cols
• Passing values as non-column

5. You have a sales DataFrame with columns: Store, Product, Month, and Revenue. You want a pivot table showing total revenue per store for each product, but only for months January and February. Which code correctly creates this pivot table?

hard

A. pd.pivot_table(df, index='Store', columns='Product', values='Revenue', aggfunc='sum').loc[['January', 'February']]

B. pd.pivot_table(df, index='Store', columns='Product', values='Revenue', aggfunc='sum', filter=['January', 'February'])

C. pd.pivot_table(df, index='Month', columns='Product', values='Revenue', aggfunc='sum').query('Month in ["January", "February"]')

D. pd.pivot_table(df[df['Month'].isin(['January', 'February'])], index='Store', columns='Product', values='Revenue', aggfunc='sum')

Solution

1. Step 1: Filter data for January and February

   Use df[df['Month'].isin(['January', 'February'])] to select only those months before pivoting.

2. Step 2: Create pivot table with Store as rows and Product as columns

   Use index='Store', columns='Product', and sum revenue with aggfunc='sum'.

3. Final Answer:

   pd.pivot_table(df[df['Month'].isin(['January', 'February'])], index='Store', columns='Product', values='Revenue', aggfunc='sum') -> Option D

4. Quick Check:

   Filter first, then pivot with sum [OK]

Hint: Filter data before pivoting to limit rows [OK]

Common Mistakes:

• Trying to filter inside pivot_table with invalid parameter
• Filtering after pivoting causing wrong results
• Using wrong index or columns