stack() and unstack() in Pandas - Time & Space Complexity

Time Complexity: stack() and unstack()

O(n x m)

Understanding Time Complexity

We want to understand how the time needed to reshape data with stack() and unstack() grows as the data size increases.

How does the work change when the data has more rows or columns?

Scenario Under Consideration

Analyze the time complexity of the following code snippet.

import pandas as pd

# Create a DataFrame with 1000 rows and 5 columns
df = pd.DataFrame({f'col{i}': range(1000) for i in range(5)})

# Use stack to reshape columns into rows
stacked = df.stack()

# Use unstack to reshape back
unstacked = stacked.unstack()

This code reshapes a DataFrame by stacking columns into a longer format and then unstacks it back to the original shape.

Identify Repeating Operations

Identify the loops, recursion, array traversals that repeat.

• Primary operation: Traversing all elements in the DataFrame to rearrange them.
• How many times: Each element is visited once during stack() and once during unstack().

How Execution Grows With Input

As the number of rows and columns grows, the total elements grow by multiplying rows and columns.

Input Size (rows x columns)	Approx. Operations
10 x 5 = 50	About 50 operations per stack or unstack
100 x 5 = 500	About 500 operations per stack or unstack
1000 x 5 = 5000	About 5000 operations per stack or unstack

Pattern observation: The work grows roughly in direct proportion to the total number of elements.

Final Time Complexity

Time Complexity: O(n x m)

This means the time needed grows linearly with the total number of elements, where n is rows and m is columns.

Common Mistake

[X] Wrong: "Stacking or unstacking only touches a few elements, so it's very fast regardless of size."

[OK] Correct: Actually, these operations must visit every element to rearrange them, so the time grows with the total data size.

Interview Connect

Knowing how reshaping data scales helps you explain your choices clearly and shows you understand the cost of data transformations.

Self-Check

"What if we stacked a DataFrame with a MultiIndex on rows? How would the time complexity change?"

Practice

1. What does the stack() function do in pandas?

easy

A. It filters rows based on a condition.

B. It moves rows into columns, making the data wider.

C. It moves columns into rows, making the data longer.

D. It sorts the DataFrame by index.

Solution

1. Step 1: Understand the purpose of stack()

   stack() takes the columns of a DataFrame and stacks them into the row index, making the data longer.

2. Step 2: Compare with other options

   unstack() does the opposite by moving rows into columns. Sorting and filtering are unrelated to stack().

3. Final Answer:

   It moves columns into rows, making the data longer. -> Option C

4. Quick Check:

   stack() = columns to rows [OK]

Hint: Remember: stack makes data longer by moving columns down [OK]

Common Mistakes:

• Confusing stack() with unstack()
• Thinking stack() sorts data
• Assuming stack() filters rows

2. Which of the following is the correct syntax to unstack a DataFrame named df?

easy

A. df.stack()

B. df.unstack()

C. df.unstacked()

D. df.stack(unstack=True)

Solution

1. Step 1: Recall the method names

   The method to move rows into columns is unstack(), so df.unstack() is correct.

2. Step 2: Check other options

   df.stack() stacks columns into rows. unstacked() does not exist. stack(unstack=True) is invalid syntax.

3. Final Answer:

   df.unstack() -> Option B

4. Quick Check:

   Unstack method is unstack() [OK]

Hint: Use df.unstack() to widen data by moving rows to columns [OK]

Common Mistakes:

• Using stack() instead of unstack()
• Calling a non-existent method unstacked()
• Passing wrong arguments to stack()

3. Given the DataFrame:

import pandas as pd
df = pd.DataFrame({
  'A': [1, 2],
  'B': [3, 4]
}, index=['x', 'y'])
stacked = df.stack()

What is the output of stacked?

medium

A. A Series with index ['A', 'B'] and values [1, 2]

B. A DataFrame with columns 'x' and 'y' and rows 'A' and 'B'

C. A DataFrame with columns 'A' and 'B' and rows 'x' and 'y'

D. A Series with MultiIndex: ('x', 'A')=1, ('x', 'B')=3, ('y', 'A')=2, ('y', 'B')=4

Solution

1. Step 1: Understand stacking on df

   Stack moves columns into rows, creating a Series with a MultiIndex combining the original index and column labels.

2. Step 2: Check the resulting Series

   The index pairs are ('x', 'A')=1, ('x', 'B')=3, ('y', 'A')=2, ('y', 'B')=4, matching A Series with MultiIndex: ('x', 'A')=1, ('x', 'B')=3, ('y', 'A')=2, ('y', 'B')=4.

3. Final Answer:

   A Series with MultiIndex: ('x', 'A')=1, ('x', 'B')=3, ('y', 'A')=2, ('y', 'B')=4 -> Option D

4. Quick Check:

   stack() output is MultiIndex Series [OK]

Hint: Stack creates MultiIndex Series combining rows and columns [OK]

Common Mistakes:

• Expecting a DataFrame output instead of Series
• Confusing index and columns after stacking
• Ignoring MultiIndex structure

4. You run the code:

df = pd.DataFrame({
  'X': [10, 20],
  'Y': [30, 40]
}, index=['a', 'b'])
result = df.unstack(level=2)

What error will occur?

medium

A. KeyError: Level 2 not found

B. TypeError: unstack() takes no arguments

C. ValueError: Cannot unstack with duplicate entries

D. No error, returns unstacked DataFrame

Solution

1. Step 1: Check the index levels of df

   df has a single index level ('a', 'b'), so level 2 does not exist.

2. Step 2: Understand unstack(level=2) behavior

   Requesting level 2 causes a KeyError because that level is missing.

3. Final Answer:

   KeyError: Level 2 not found -> Option A

4. Quick Check:

   unstack(level) must match existing index levels [OK]

Hint: Check index levels before unstacking by level number [OK]

Common Mistakes:

• Assuming unstack() accepts any level number
• Expecting no error on invalid level
• Confusing level number with label

5. You have a DataFrame with a MultiIndex on rows:

import pandas as pd
index = pd.MultiIndex.from_tuples([('A', 1), ('A', 2), ('B', 1)])
df = pd.DataFrame({'val': [10, 20, 30]}, index=index)

How can you reshape df so that the second level of the index becomes columns?

hard

A. Use df.unstack(level=1) to move second index level to columns.

B. Use df.stack(level=1) to move second index level to columns.

C. Use df.unstack(level=0) to move second index level to columns.

D. Use df.stack() without arguments.

Solution

1. Step 1: Identify the MultiIndex levels

   The DataFrame has two index levels: level 0 ('A', 'B') and level 1 (1, 2).

2. Step 2: Use unstack(level=1) to move second level to columns

   Unstacking level 1 moves that index level into columns, reshaping the data as required.

3. Final Answer:

   Use df.unstack(level=1) to move second index level to columns. -> Option A

4. Quick Check:

   unstack(level=1) moves second index level to columns [OK]

Hint: Unstack the desired index level number to widen data [OK]

Common Mistakes:

• Using stack() instead of unstack()
• Unstacking wrong level number
• Ignoring MultiIndex structure