concat() for stacking DataFrames in Pandas - Time & Space Complexity

Time Complexity: concat() for stacking DataFrames

O(n)

Understanding Time Complexity

When stacking DataFrames using pandas concat(), it is important to understand how the time needed grows as the data gets bigger.

We want to know how the work done changes when we add more rows or columns.

Scenario Under Consideration

Analyze the time complexity of the following code snippet.

import pandas as pd

df1 = pd.DataFrame({'A': range(1000), 'B': range(1000)})
df2 = pd.DataFrame({'A': range(1000, 2000), 'B': range(1000, 2000)})

result = pd.concat([df1, df2], axis=0, ignore_index=True)

This code stacks two DataFrames vertically, combining their rows into one larger DataFrame.

Identify Repeating Operations

Identify the loops, recursion, array traversals that repeat.

• Primary operation: Copying rows from each DataFrame into a new combined DataFrame.
• How many times: Once for each row in both DataFrames, so total rows combined.

How Execution Grows With Input

As the number of rows increases, the time to copy and combine grows roughly in direct proportion.

Input Size (n)	Approx. Operations
10	About 20 row copies
100	About 200 row copies
1000	About 2000 row copies

Pattern observation: Doubling the input roughly doubles the work done.

Final Time Complexity

Time Complexity: O(n)

This means the time needed grows linearly with the total number of rows being stacked.

Common Mistake

[X] Wrong: "Concatenating two DataFrames is a constant time operation regardless of size."

[OK] Correct: The operation must copy all rows to create a new DataFrame, so time grows with the total rows.

Interview Connect

Understanding how concat() scales helps you reason about data processing speed and memory use in real projects.

Self-Check

"What if we stacked 10 DataFrames instead of 2? How would the time complexity change?"

Practice

1. What does the concat() function do in pandas?

easy

A. Deletes rows from a DataFrame

B. Joins DataFrames by stacking rows or columns

C. Sorts the DataFrame by index

D. Filters rows based on a condition

Solution

1. Step 1: Understand the purpose of concat()

   concat() is used to combine multiple DataFrames either by stacking rows or columns.

2. Step 2: Compare with other DataFrame operations

   Deleting, sorting, or filtering are different operations and not related to concat().

3. Final Answer:

   Joins DataFrames by stacking rows or columns -> Option B

4. Quick Check:

   concat() stacks DataFrames [OK]

Hint: Remember: concat stacks rows or columns [OK]

Common Mistakes:

• Confusing concat with filtering or sorting
• Thinking concat deletes data
• Assuming concat only merges columns

2. Which of the following is the correct syntax to stack two DataFrames df1 and df2 vertically using concat()?

easy

A. pd.concat(df1 + df2)

B. pd.concat([df1, df2], axis=1)

C. pd.concat(df1, df2)

D. pd.concat([df1, df2], axis=0)

Solution

1. Step 1: Identify axis for vertical stacking

   Vertical stacking means stacking rows, which is done with axis=0.

2. Step 2: Check syntax correctness

   pd.concat() requires a list of DataFrames as first argument, so [df1, df2] is correct.

3. Final Answer:

   pd.concat([df1, df2], axis=0) -> Option D

4. Quick Check:

   Vertical stack uses axis=0 [OK]

Hint: Use axis=0 to stack rows vertically [OK]

Common Mistakes:

• Using axis=1 for vertical stacking
• Passing DataFrames without a list
• Trying to add DataFrames with + inside concat

3. Given the following code:

import pandas as pd
df1 = pd.DataFrame({'A': [1, 2], 'B': [3, 4]})
df2 = pd.DataFrame({'A': [5, 6], 'B': [7, 8]})
result = pd.concat([df1, df2], ignore_index=True)
print(result)

What is the output?

medium

A. DataFrame with 4 rows, original indexes 0,1,0,1

B. DataFrame with 2 rows, columns stacked horizontally

C. DataFrame with 4 rows, index reset from 0 to 3

D. Error because ignore_index is invalid

Solution

1. Step 1: Understand concat with ignore_index=True

   Concatenating df1 and df2 vertically stacks rows, resulting in 4 rows total. ignore_index=True resets the index to 0,1,2,3.

2. Step 2: Confirm output structure

   The columns remain the same ('A' and 'B'), rows are stacked, and index is reset.

3. Final Answer:

   DataFrame with 4 rows, index reset from 0 to 3 -> Option C

4. Quick Check:

   ignore_index=True resets index [OK]

Hint: ignore_index=True resets row numbers after stacking [OK]

Common Mistakes:

• Forgetting ignore_index resets index
• Thinking columns stack horizontally by default
• Assuming error with ignore_index parameter

4. What is wrong with this code snippet?

import pandas as pd
df1 = pd.DataFrame({'X': [1, 2]})
df2 = pd.DataFrame({'Y': [3, 4]})
result = pd.concat(df1, df2, axis=0)
print(result)

medium

A. pd.concat requires a list of DataFrames as first argument

B. axis=0 stacks columns instead of rows

C. DataFrames must have same columns to concat

D. ignore_index=True is missing

Solution

1. Step 1: Check pd.concat argument format

   pd.concat() expects a list or iterable of DataFrames as the first argument, but here two DataFrames are passed as separate arguments.

2. Step 2: Identify correct usage

   The correct call is pd.concat([df1, df2], axis=0) with a list.

3. Final Answer:

   pd.concat requires a list of DataFrames as first argument -> Option A

4. Quick Check:

   pd.concat([df1, df2]) needs list [OK]

Hint: Always pass DataFrames as a list to concat [OK]

Common Mistakes:

• Passing DataFrames as separate arguments
• Confusing axis parameter meaning
• Assuming columns must match exactly

5. You have two DataFrames:

df1 = pd.DataFrame({'A': [1, 2], 'B': [3, 4]})
df2 = pd.DataFrame({'B': [5, 6], 'C': [7, 8]})

You want to stack them vertically and keep all columns, filling missing values with NaN. Which code will do this correctly?

hard

A. pd.concat([df1, df2], axis=0, ignore_index=True)

B. pd.concat([df1, df2], axis=1)

C. pd.concat([df1, df2], axis=0).fillna(0)

D. pd.concat([df1, df2], axis=0, join='inner')

Solution

1. Step 1: Understand vertical stacking with different columns

   Stacking vertically with axis=0 combines rows. Since columns differ, missing columns get NaN by default.

2. Step 2: Check parameters for desired behavior

   ignore_index=True resets index. Default join='outer' keeps all columns and fills missing with NaN.

3. Step 3: Evaluate other options

   pd.concat([df1, df2], axis=1) stacks columns horizontally, not vertically. pd.concat([df1, df2], axis=0).fillna(0) fills NaN with 0, not requested. pd.concat([df1, df2], axis=0, join='inner') uses join='inner' which keeps only common columns.

4. Final Answer:

   pd.concat([df1, df2], axis=0, ignore_index=True) -> Option A

5. Quick Check:

   axis=0 + default join='outer' keeps all columns [OK]

Hint: Use axis=0 and default join to keep all columns [OK]

Common Mistakes:

• Using axis=1 to stack rows
• Using join='inner' and losing columns
• Filling NaN when not needed