duplicated() for finding duplicates in Pandas - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of duplicated() with default parameters

What is the output of the following code snippet?

Pandas

import pandas as pd

df = pd.DataFrame({'A': [1, 2, 2, 3, 4, 4, 4], 'B': ['x', 'y', 'y', 'z', 'x', 'x', 'x']})
result = df.duplicated()
print(result.tolist())

A [True, False, True, False, True, False, True]

B [False, True, True, False, True, True, True]

C [False, False, False, False, False, False, False]

D [False, False, True, False, False, True, True]

💡 Hint

duplicated() marks all rows that have appeared before as True, except the first occurrence.

The duplicated() method by default marks all rows that are duplicates of a previous row as True. The first occurrence is marked False.

In the DataFrame, the second '2' in column A is a duplicate, so True. The second and third '4's are duplicates, so True.

❓ data_output

intermediate

Count of duplicated rows with subset and keep='last'

Given the DataFrame below, how many rows are marked as duplicates when using subset=['A'] and keep='last'?

Pandas

import pandas as pd

df = pd.DataFrame({'A': [5, 6, 5, 7, 6, 8, 5]})
duplicates = df.duplicated(subset=['A'], keep='last')
count = duplicates.sum()
print(count)

A 3

B 2

C 1

D 4

💡 Hint

keep='last' marks all duplicates except the last occurrence as True.

Values 5 and 6 appear multiple times. With keep='last', the first occurrences are marked True (duplicates), and the last occurrence is False.

For 5: positions 0 and 2 are duplicates (True), position 6 is last (False). For 6: position 1 is duplicate (True), position 4 is last (False). Total duplicates: 3.

🔧 Debug

advanced

Identify the error in duplicated() usage

What error will this code raise?

Pandas

import pandas as pd

df = pd.DataFrame({'A': [1, 2, 3]})
result = df.duplicated(keep='middle')
print(result)

A ValueError: keep must be either 'first', 'last', or False

B TypeError: duplicated() got an unexpected keyword argument 'keep'

C SyntaxError: invalid syntax

D No error, prints a boolean Series

💡 Hint

Check the allowed values for the keep parameter in duplicated().

The keep parameter only accepts 'first', 'last', or False. 'middle' is invalid and causes a ValueError.

🚀 Application

advanced

Filter unique rows using duplicated()

Which code snippet correctly filters the DataFrame to keep only unique rows (no duplicates) based on all columns?

Pandas

import pandas as pd

df = pd.DataFrame({'A': [1, 2, 2, 3], 'B': ['a', 'b', 'b', 'c']})

A df[df.duplicated(keep=False)]

B df[df.duplicated(keep='first')]

C df[~df.duplicated(keep=False)]

D df[~df.duplicated(keep='last')]

💡 Hint

Use keep=False to mark all duplicates as True, then invert to get unique rows.

duplicated(keep=False) marks all duplicates as True. Using ~ inverts the mask to select only unique rows.

Option C correctly filters unique rows.

🧠 Conceptual

expert

Understanding duplicated() behavior with subset and keep=False

Consider a DataFrame with columns 'X' and 'Y'. If you run df.duplicated(subset=['X'], keep=False), what does the output represent?

A Marks only the first occurrence of each duplicate value in column 'X' as True.

B Marks all rows that have duplicate values in column 'X', including the first and last occurrences.

C Marks only the last occurrence of each duplicate value in column 'X' as True.

D Marks no rows as duplicates because keep=False disables duplicate detection.

💡 Hint

keep=False means no row is considered unique among duplicates.

With keep=False, duplicated() marks all duplicates as True, including the first and last occurrences.

Using subset=['X'] limits the check to column 'X'.

Practice

1. What does the duplicated() function in pandas do?

easy

A. It identifies duplicate rows in a DataFrame.

B. It removes duplicate rows from a DataFrame.

C. It sorts the DataFrame by duplicate values.

D. It counts the number of unique rows in a DataFrame.

Solution

1. Step 1: Understand the purpose of duplicated()

   The duplicated() function checks each row and marks it as True if it is a duplicate of a previous row.

2. Step 2: Differentiate from other functions

   Unlike functions that remove duplicates or count unique rows, duplicated() only flags duplicates without changing the DataFrame.

3. Final Answer:

   It identifies duplicate rows in a DataFrame. -> Option A

4. Quick Check:

   duplicated() = identifies duplicates [OK]

Hint: Remember: duplicated() flags duplicates, doesn't remove them [OK]

Common Mistakes:

• Confusing duplicated() with drop_duplicates()
• Thinking duplicated() sorts data
• Assuming duplicated() counts unique rows

2. Which of the following is the correct syntax to find duplicates in a pandas DataFrame df?

easy

A. df.duplicated()

B. df.duplicates()

C. df.find_duplicates()

D. df.duplicate()

Solution

1. Step 1: Recall the exact function name

   The correct pandas function to find duplicate rows is duplicated(), not any other variation.

2. Step 2: Verify syntax correctness

   The function is called on the DataFrame object as df.duplicated() with parentheses.

3. Final Answer:

   df.duplicated() -> Option A

4. Quick Check:

   Correct function name = df.duplicated() [OK]

Hint: Use exact function name: duplicated() with parentheses [OK]

Common Mistakes:

• Misspelling the function name
• Omitting parentheses after duplicated
• Using non-existent functions like duplicates()

3. Given the DataFrame df below, what is the output of df.duplicated()?

import pandas as pd
data = {'A': [1, 2, 2, 3], 'B': ['x', 'y', 'y', 'z']}
df = pd.DataFrame(data)
print(df.duplicated())

medium

A. [True, False, True, False]

B. [False, False, True, False]

C. [False, True, False, True]

D. [False, False, False, True]

Solution

1. Step 1: Identify duplicate rows in the DataFrame

   The rows are: 0: (1, 'x'), 1: (2, 'y'), 2: (2, 'y'), 3: (3, 'z'). Row 2 is a duplicate of row 1.

2. Step 2: Understand duplicated() output

   duplicated() returns False for the first occurrence and True for duplicates. So, rows 0,1,3 are False; row 2 is True.

3. Final Answer:

   [False, False, True, False] -> Option B

4. Quick Check:

   First duplicate row marked True = [False, False, True, False] [OK]

Hint: First occurrence is False, duplicates after are True [OK]

Common Mistakes:

• Marking first occurrence as True
• Confusing row indices with values
• Assuming all duplicates are True regardless of position

4. What is wrong with this code snippet to find duplicates in column 'A' of DataFrame df?

duplicates = df.duplicated['A']
print(duplicates)

medium

A. duplicated() cannot check specific columns

B. Missing import statement for pandas

C. The print statement is incorrect

D. Using square brackets instead of parentheses to call duplicated()

Solution

1. Step 1: Check function call syntax

   The code uses square brackets [] instead of parentheses () to call the duplicated() method, which is incorrect syntax.

2. Step 2: Correct way to specify columns

   To check duplicates based on column 'A', the correct syntax is df.duplicated(subset=['A']) with parentheses.

3. Final Answer:

   Using square brackets instead of parentheses to call duplicated() -> Option D

4. Quick Check:

   Call methods with () not [] [OK]

Hint: Call duplicated() with parentheses, not square brackets [OK]

Common Mistakes:

• Using [] instead of () to call functions
• Trying to access duplicated as an attribute
• Not using subset parameter for specific columns

5. You have a DataFrame df with columns 'Name', 'Age', and 'City'. You want to find duplicates only based on 'Name' and 'City', ignoring 'Age'. Which code correctly finds these duplicates?

hard

A. df.duplicated(['Name', 'City'], keep=True)

B. df.duplicated(columns=['Name', 'City'])

C. df.duplicated(subset=['Name', 'City'])

D. df.duplicated('Name', 'City')

Solution

1. Step 1: Use subset parameter to specify columns

   To check duplicates based on specific columns, use the subset parameter with a list of column names.

2. Step 2: Validate correct syntax

   The correct syntax is df.duplicated(subset=['Name', 'City']). df.duplicated(columns=['Name', 'City']) uses a wrong parameter name (columns), and options A and C use invalid syntax.

3. Final Answer:

   df.duplicated(subset=['Name', 'City']) -> Option C

4. Quick Check:

   Use subset=['col1', 'col2'] to check specific columns [OK]

Hint: Use subset parameter to check duplicates on chosen columns [OK]

Common Mistakes:

• Using wrong parameter name like columns instead of subset
• Passing list without parameter name
• Confusing keep parameter with subset