duplicated() for finding duplicates in Pandas - Time & Space Complexity

Time Complexity: duplicated() for finding duplicates

O(n)

Understanding Time Complexity

We want to understand how the time needed to find duplicates grows as the data gets bigger.

How does pandas check each row to see if it appeared before?

Scenario Under Consideration

Analyze the time complexity of the following code snippet.

import pandas as pd

data = pd.DataFrame({
    'A': [1, 2, 2, 3, 4, 4, 4],
    'B': ['x', 'y', 'y', 'z', 'x', 'x', 'x']
})

duplicates = data.duplicated()
print(duplicates)

This code creates a small table and finds which rows are duplicates of earlier rows.

Identify Repeating Operations

Identify the loops, recursion, array traversals that repeat.

• Primary operation: Hashing each row and checking if it was seen before in a dictionary.
• How many times: Once for each row (O(1) average time per check).

How Execution Grows With Input

As the number of rows grows, the work to find duplicates grows in proportion to the number of rows.

Input Size (n)	Approx. Operations
10	About 10 operations
100	About 100 operations
1000	About 1,000 operations

Pattern observation: Doubling the number of rows roughly doubles the work needed.

Final Time Complexity

Time Complexity: O(n)

This means the time to find duplicates grows linearly as the data gets bigger, thanks to efficient hashing.

Common Mistake

[X] Wrong: "Finding duplicates requires checking each row against all previous rows, so O(n²)."

[OK] Correct: Pandas uses hashing (dictionary lookups) for O(1) average checks per row, making it O(n) total.

Interview Connect

Understanding how duplicate detection scales helps you explain data processing limits and choose better methods when needed.

Self-Check

"What if we only checked duplicates in one column instead of all columns? How would the time complexity change?"

Practice

1. What does the duplicated() function in pandas do?

easy

A. It identifies duplicate rows in a DataFrame.

B. It removes duplicate rows from a DataFrame.

C. It sorts the DataFrame by duplicate values.

D. It counts the number of unique rows in a DataFrame.

Solution

1. Step 1: Understand the purpose of duplicated()

   The duplicated() function checks each row and marks it as True if it is a duplicate of a previous row.

2. Step 2: Differentiate from other functions

   Unlike functions that remove duplicates or count unique rows, duplicated() only flags duplicates without changing the DataFrame.

3. Final Answer:

   It identifies duplicate rows in a DataFrame. -> Option A

4. Quick Check:

   duplicated() = identifies duplicates [OK]

Hint: Remember: duplicated() flags duplicates, doesn't remove them [OK]

Common Mistakes:

• Confusing duplicated() with drop_duplicates()
• Thinking duplicated() sorts data
• Assuming duplicated() counts unique rows

2. Which of the following is the correct syntax to find duplicates in a pandas DataFrame df?

easy

A. df.duplicated()

B. df.duplicates()

C. df.find_duplicates()

D. df.duplicate()

Solution

1. Step 1: Recall the exact function name

   The correct pandas function to find duplicate rows is duplicated(), not any other variation.

2. Step 2: Verify syntax correctness

   The function is called on the DataFrame object as df.duplicated() with parentheses.

3. Final Answer:

   df.duplicated() -> Option A

4. Quick Check:

   Correct function name = df.duplicated() [OK]

Hint: Use exact function name: duplicated() with parentheses [OK]

Common Mistakes:

• Misspelling the function name
• Omitting parentheses after duplicated
• Using non-existent functions like duplicates()

3. Given the DataFrame df below, what is the output of df.duplicated()?

import pandas as pd
data = {'A': [1, 2, 2, 3], 'B': ['x', 'y', 'y', 'z']}
df = pd.DataFrame(data)
print(df.duplicated())

medium

A. [True, False, True, False]

B. [False, False, True, False]

C. [False, True, False, True]

D. [False, False, False, True]

Solution

1. Step 1: Identify duplicate rows in the DataFrame

   The rows are: 0: (1, 'x'), 1: (2, 'y'), 2: (2, 'y'), 3: (3, 'z'). Row 2 is a duplicate of row 1.

2. Step 2: Understand duplicated() output

   duplicated() returns False for the first occurrence and True for duplicates. So, rows 0,1,3 are False; row 2 is True.

3. Final Answer:

   [False, False, True, False] -> Option B

4. Quick Check:

   First duplicate row marked True = [False, False, True, False] [OK]

Hint: First occurrence is False, duplicates after are True [OK]

Common Mistakes:

• Marking first occurrence as True
• Confusing row indices with values
• Assuming all duplicates are True regardless of position

4. What is wrong with this code snippet to find duplicates in column 'A' of DataFrame df?

duplicates = df.duplicated['A']
print(duplicates)

medium

A. duplicated() cannot check specific columns

B. Missing import statement for pandas

C. The print statement is incorrect

D. Using square brackets instead of parentheses to call duplicated()

Solution

1. Step 1: Check function call syntax

   The code uses square brackets [] instead of parentheses () to call the duplicated() method, which is incorrect syntax.

2. Step 2: Correct way to specify columns

   To check duplicates based on column 'A', the correct syntax is df.duplicated(subset=['A']) with parentheses.

3. Final Answer:

   Using square brackets instead of parentheses to call duplicated() -> Option D

4. Quick Check:

   Call methods with () not [] [OK]

Hint: Call duplicated() with parentheses, not square brackets [OK]

Common Mistakes:

• Using [] instead of () to call functions
• Trying to access duplicated as an attribute
• Not using subset parameter for specific columns

5. You have a DataFrame df with columns 'Name', 'Age', and 'City'. You want to find duplicates only based on 'Name' and 'City', ignoring 'Age'. Which code correctly finds these duplicates?

hard

A. df.duplicated(['Name', 'City'], keep=True)

B. df.duplicated(columns=['Name', 'City'])

C. df.duplicated(subset=['Name', 'City'])

D. df.duplicated('Name', 'City')

Solution

1. Step 1: Use subset parameter to specify columns

   To check duplicates based on specific columns, use the subset parameter with a list of column names.

2. Step 2: Validate correct syntax

   The correct syntax is df.duplicated(subset=['Name', 'City']). df.duplicated(columns=['Name', 'City']) uses a wrong parameter name (columns), and options A and C use invalid syntax.

3. Final Answer:

   df.duplicated(subset=['Name', 'City']) -> Option C

4. Quick Check:

   Use subset=['col1', 'col2'] to check specific columns [OK]

Hint: Use subset parameter to check duplicates on chosen columns [OK]

Common Mistakes:

• Using wrong parameter name like columns instead of subset
• Passing list without parameter name
• Confusing keep parameter with subset