Why combining DataFrames matters in Pandas - Performance Analysis

Time Complexity: Why combining DataFrames matters

O(n)

Understanding Time Complexity

When we combine DataFrames, the time it takes depends on how much data we have. We want to understand how this time grows as the data gets bigger.

How does the work needed change when we join or merge tables?

Scenario Under Consideration

Analyze the time complexity of the following code snippet.

import pandas as pd

n = 10  # example size

df1 = pd.DataFrame({"key": range(n), "value1": range(n)})
df2 = pd.DataFrame({"key": range(n), "value2": range(n)})

result = pd.merge(df1, df2, on="key")

This code merges two DataFrames on a common column called "key".

Identify Repeating Operations

Identify the loops, recursion, array traversals that repeat.

• Primary operation: Matching rows from both DataFrames by the "key" column.
• How many times: Each row in the first DataFrame is compared to rows in the second DataFrame to find matches.

How Execution Grows With Input

As the number of rows (n) grows, the work to find matching keys grows too.

Input Size (n)	Approx. Operations
10	About 10 matching checks
100	About 100 matching checks
1000	About 1000 matching checks

Pattern observation: The work grows roughly in direct proportion to the number of rows.

Final Time Complexity

Time Complexity: O(n)

This means the time to combine DataFrames grows linearly with the number of rows.

Common Mistake

[X] Wrong: "Merging two DataFrames always takes a long time because it compares every row to every other row."

[OK] Correct: Pandas uses efficient methods like hashing or sorting to avoid checking every pair, so it usually works in linear time, not quadratic.

Interview Connect

Understanding how combining data grows with size helps you explain your approach clearly and shows you know how to handle real data efficiently.

Self-Check

"What if we merged on multiple columns instead of one? How would the time complexity change?"

Practice

1. Why do we combine DataFrames in pandas?
pd.concat and pd.merge are common methods. What is the main reason to combine DataFrames?

easy

A. To get a complete view by joining or stacking data

B. To delete duplicate rows automatically

C. To convert DataFrames into lists

D. To change the data types of columns

Solution

1. Step 1: Understand the purpose of combining DataFrames

   Combining DataFrames helps to bring together data from different sources or parts to analyze them as one.

2. Step 2: Identify the main reason

   Using pd.concat stacks data vertically or horizontally, and pd.merge joins data based on matching columns, both to get a fuller dataset.

3. Final Answer:

   To get a complete view by joining or stacking data -> Option A

4. Quick Check:

   Combining DataFrames = Complete view [OK]

Hint: Combining means joining or stacking data for full info [OK]

Common Mistakes:

• Thinking combining deletes duplicates automatically
• Confusing combining with data type changes
• Believing combining converts DataFrames to lists

2. Which of the following is the correct syntax to stack two DataFrames df1 and df2 vertically using pandas?

easy

A. pd.merge(df1, df2, axis=0)

B. pd.concat([df1, df2], axis=0)

C. df1.append(df2, axis=1)

D. pd.concat(df1, df2, axis=1)

Solution

1. Step 1: Recall the function to stack DataFrames vertically

   pd.concat stacks DataFrames; axis=0 means stacking rows vertically.

2. Step 2: Check the syntax

   The correct syntax is pd.concat([df1, df2], axis=0). Other options misuse pd.merge or wrong axis or argument types.

3. Final Answer:

   pd.concat([df1, df2], axis=0) -> Option B

4. Quick Check:

   Stack vertically = concat with axis=0 [OK]

Hint: Use pd.concat with a list and axis=0 to stack rows [OK]

Common Mistakes:

• Using pd.merge instead of pd.concat for stacking
• Passing DataFrames without a list to pd.concat
• Using wrong axis value for vertical stacking

3. What will be the output of this code?

import pandas as pd

df1 = pd.DataFrame({'key': ['A', 'B'], 'val1': [1, 2]})
df2 = pd.DataFrame({'key': ['B', 'C'], 'val2': [3, 4]})
result = pd.merge(df1, df2, on='key', how='inner')
print(result)

medium

A. key val1
0 A 1
1 B 2

B. key val1 val2
0 A 1 3
1 B 2 4

C. key val1 val2
0 A 1 NaN
1 B 2 3
2 C NaN 4

D. key val1 val2
0 B 2 3

Solution

1. Step 1: Understand pd.merge with how='inner'

   Inner join keeps only rows with keys present in both DataFrames. Here keys are 'A', 'B' in df1 and 'B', 'C' in df2.

2. Step 2: Find common keys and merge rows

   Only 'B' is common. So result has one row with key 'B', val1=2 from df1, val2=3 from df2.

3. Final Answer:

   key val1 val2 0 B 2 3 -> Option D

4. Quick Check:

   Inner join = common keys only [OK]

Hint: Inner merge keeps only matching keys in both DataFrames [OK]

Common Mistakes:

• Expecting all keys from both DataFrames (confusing with outer join)
• Mixing up val2 values for keys
• Ignoring how='inner' effect

4. The following code throws an error. What is the main issue?

import pandas as pd

df1 = pd.DataFrame({'A': [1, 2]})
df2 = pd.DataFrame({'B': [3, 4]})
result = pd.merge(df1, df2, on='A')
print(result)

medium

A. Column 'A' does not exist in both DataFrames

B. pd.merge cannot merge DataFrames with different columns

C. pd.merge requires axis parameter

D. DataFrames must have same number of rows to merge

Solution

1. Step 1: Check columns in both DataFrames

   df1 has column 'A', but df2 has only column 'B'. So 'A' is missing in df2.

2. Step 2: Understand pd.merge on parameter

   pd.merge requires the 'on' column to be present in both DataFrames to join on it. Missing column causes error.

3. Final Answer:

   Column 'A' does not exist in both DataFrames -> Option A

4. Quick Check:

   Merge 'on' column must be in both DataFrames [OK]

Hint: Merge 'on' column must exist in both DataFrames [OK]

Common Mistakes:

• Thinking axis parameter is needed for merge
• Believing DataFrames must have same row count
• Assuming merge works with different columns without 'on'

5. You have two DataFrames:

df1 = pd.DataFrame({'id': [1, 2, 3], 'score': [90, 80, 70]})
df2 = pd.DataFrame({'id': [2, 3, 4], 'grade': ['B', 'C', 'A']})

You want a DataFrame that includes all ids from both, with scores and grades matched where possible, and NaN where missing. Which code achieves this?

hard

A. pd.concat([df1, df2], axis=0)

B. pd.concat([df1, df2], axis=1)

C. pd.merge(df1, df2, on='id', how='outer')

D. pd.merge(df1, df2, on='id', how='inner')

Solution

1. Step 1: Understand the goal

   We want all ids from both DataFrames, matching scores and grades where ids overlap, and NaN where data is missing.

2. Step 2: Choose the correct merge type

   Using how='outer' in pd.merge keeps all keys from both DataFrames, filling missing values with NaN.

3. Step 3: Check other options

   pd.concat stacks data but does not join on keys; how='inner' drops ids not in both.

4. Final Answer:

   pd.merge(df1, df2, on='id', how='outer') -> Option C

5. Quick Check:

   Outer merge = all keys, fill missing with NaN [OK]

Hint: Use outer merge to keep all keys and fill missing data [OK]

Common Mistakes:

• Using inner merge and losing unmatched ids
• Using concat which stacks instead of joining
• Confusing axis parameter with merge how