concat() for stacking DataFrames in Data Analysis Python - Time & Space Complexity

Time Complexity: concat() for stacking DataFrames

O(n)

Understanding Time Complexity

When stacking data tables using concat(), it is important to know how the time needed grows as the data gets bigger.

We want to find out how the work done changes when we add more rows or columns.

Scenario Under Consideration

Analyze the time complexity of the following code snippet.

import pandas as pd

n = 10  # Example value for n

df1 = pd.DataFrame({'A': range(n)})
df2 = pd.DataFrame({'A': range(n, 2*n)})

result = pd.concat([df1, df2], axis=0, ignore_index=True)

This code stacks two data tables one below the other, combining their rows into one table.

Identify Repeating Operations

Identify the loops, recursion, array traversals that repeat.

• Primary operation: Copying rows from each DataFrame into a new combined DataFrame.
• How many times: Each row from both tables is processed once, so about 2*n times if each has n rows.

How Execution Grows With Input

As the number of rows grows, the work grows roughly in direct proportion.

Input Size (n)	Approx. Operations
10	About 20 row copies
100	About 200 row copies
1000	About 2000 row copies

Pattern observation: Doubling the rows doubles the work needed.

Final Time Complexity

Time Complexity: O(n)

This means the time to stack grows linearly with the total number of rows.

Common Mistake

[X] Wrong: "concat() is instant no matter how big the data is."

[OK] Correct: The function must copy all rows to create the new table, so bigger data takes more time.

Interview Connect

Understanding how concat() scales helps you explain data merging choices clearly and shows you know how data size affects performance.

Self-Check

"What if we stacked 10 DataFrames instead of 2? How would the time complexity change?"

Practice

1. What does the concat() function do when stacking DataFrames with axis=0?

easy

A. It sorts the DataFrames before stacking.

B. It stacks DataFrames horizontally by adding columns.

C. It stacks DataFrames vertically by adding rows.

D. It merges DataFrames based on matching column values.

Solution

1. Step 1: Understand axis parameter in concat()

   The axis=0 means stacking along rows, so new rows are added below existing ones.

2. Step 2: Differentiate vertical vs horizontal stacking

   Vertical stacking adds rows; horizontal stacking (axis=1) adds columns side by side.

3. Final Answer:

   It stacks DataFrames vertically by adding rows. -> Option C

4. Quick Check:

   axis=0 means vertical stacking [OK]

Hint: axis=0 adds rows, axis=1 adds columns [OK]

Common Mistakes:

• Confusing axis=0 with horizontal stacking
• Thinking concat merges like join
• Assuming concat sorts data automatically

2. Which of the following is the correct syntax to stack two DataFrames df1 and df2 vertically and reset the index?

easy

A. pd.concat([df1, df2], axis=1, ignore_index=True)

B. pd.concat([df1, df2], axis=0, ignore_index=True)

C. pd.concat(df1, df2, axis=0, ignore_index=True)

D. pd.concat([df1, df2], axis=0)

Solution

1. Step 1: Check the concat() function syntax

   Correct syntax requires a list of DataFrames as first argument, e.g., [df1, df2].

2. Step 2: Confirm axis and ignore_index usage

   To stack vertically and reset index, use axis=0 and ignore_index=True.

3. Final Answer:

   pd.concat([df1, df2], axis=0, ignore_index=True) -> Option B

4. Quick Check:

   List + axis=0 + ignore_index=True = correct syntax [OK]

Hint: Use list of DataFrames and axis=0 for vertical concat [OK]

Common Mistakes:

• Passing DataFrames without list brackets
• Using axis=1 for vertical stacking
• Forgetting ignore_index to reset row numbers

3. Given these DataFrames:

import pandas as pd
df1 = pd.DataFrame({'A': [1, 2], 'B': [3, 4]})
df2 = pd.DataFrame({'A': [5], 'B': [6]})
result = pd.concat([df1, df2], axis=0, ignore_index=True)
print(result)

What is the output?

medium

A. A B
0 1 3
1 2 4
2 5 6

B. A B
0 1 3
1 2 4
0 5 6

C. A B
0 1 3
1 2 4

D. Error due to missing axis parameter

Solution

1. Step 1: Understand concat with axis=0 and ignore_index=True

   Rows from df2 are added below df1, and index resets from 0 to 2.

2. Step 2: Combine rows and reset index

   Result has 3 rows with index 0,1,2 and columns A and B filled accordingly.

3. Final Answer:

   A B 0 1 3 1 2 4 2 5 6 -> Option A

4. Quick Check:

   concat vertical + ignore_index resets index [OK]

Hint: ignore_index=True resets row numbers after concat [OK]

Common Mistakes:

• Expecting original index to remain
• Confusing axis=0 with axis=1 output
• Forgetting to pass DataFrames as list

4. What is wrong with this code snippet?

import pandas as pd
df1 = pd.DataFrame({'X': [1, 2]})
df2 = pd.DataFrame({'Y': [3, 4]})
result = pd.concat(df1, df2, axis=0)
print(result)

medium

A. Missing list brackets around DataFrames in concat.

B. axis=0 is invalid for concat.

C. DataFrames must have same columns to concat.

D. ignore_index=True is required.

Solution

1. Step 1: Check concat() argument format

   concat() expects a list of DataFrames as first argument, but here two DataFrames are passed separately.

2. Step 2: Identify correct usage

   Correct call is pd.concat([df1, df2], axis=0) with brackets around DataFrames.

3. Final Answer:

   Missing list brackets around DataFrames in concat. -> Option A

4. Quick Check:

   concat needs list of DataFrames [OK]

Hint: Always pass DataFrames as a list to concat() [OK]

Common Mistakes:

• Passing DataFrames as separate arguments
• Assuming axis=0 is invalid
• Thinking ignore_index is mandatory

5. You have two DataFrames:

df1 = pd.DataFrame({'Name': ['Alice', 'Bob'], 'Score': [85, 90]})
df2 = pd.DataFrame({'Name': ['Charlie'], 'Score': [88]})

You want to stack them vertically but keep the original index from both DataFrames. Which code will do this correctly?

hard

A. pd.concat([df1, df2], axis=0, ignore_index=True)

B. pd.concat(df1, df2, axis=0)

C. pd.concat([df1, df2], axis=1)

D. pd.concat([df1, df2], axis=0)

Solution

1. Step 1: Understand index behavior in concat()

   By default, concat keeps original indexes unless ignore_index=True is set.

2. Step 2: Choose correct parameters

   To stack vertically and keep indexes, use axis=0 without ignore_index.

3. Final Answer:

   pd.concat([df1, df2], axis=0) -> Option D

4. Quick Check:

   Default concat keeps original index [OK]

Hint: Omit ignore_index to keep original indexes [OK]

Common Mistakes:

• Using ignore_index=True when original index is needed
• Using axis=1 for vertical stacking
• Passing DataFrames without list brackets