Counting duplicates in Pandas - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Counting duplicate rows in a DataFrame

What is the output of this code that counts duplicate rows in a pandas DataFrame?

Pandas

import pandas as pd

df = pd.DataFrame({
    'A': [1, 2, 2, 3, 3, 3],
    'B': ['x', 'y', 'y', 'z', 'z', 'z']
})

count_duplicates = df.duplicated().sum()
print(count_duplicates)

A 4

B 2

C 0

D 3

💡 Hint

Remember that duplicated() marks all rows except the first occurrence as duplicates.

The DataFrame has 6 rows. Rows 2, 4, and 5 are duplicates of previous rows, so duplicated() returns True for these 3 rows. Summing True values gives 3.

❓ data_output

intermediate

Counting duplicates with subset columns

Given this DataFrame, what is the output of counting duplicates only based on column 'A'?

Pandas

import pandas as pd

df = pd.DataFrame({
    'A': [1, 2, 2, 3, 3, 3],
    'B': ['x', 'y', 'z', 'z', 'y', 'z']
})

count_dup_subset = df.duplicated(subset=['A']).sum()
print(count_dup_subset)

A 3

B 2

C 1

D 0

💡 Hint

Duplicates are counted only by column 'A', ignoring column 'B'.

Column 'A' has values [1, 2, 2, 3, 3, 3]. The first occurrences are at indices 0,1,3. The duplicates are at indices 2,4,5. So total duplicates are 3. But duplicated() marks only the second and later duplicates, so indices 2,4,5 are True. Summing gives 3.

🔧 Debug

advanced

Identify the error in counting duplicates

What error does this code raise when trying to count duplicates in a DataFrame?

Pandas

import pandas as pd

df = pd.DataFrame({'A': [1, 2, 2], 'B': ['x', 'y', 'y']})

count = df.duplicated(subset='A', keep='maybe').sum()
print(count)

A ValueError: keep must be one of {'first', 'last', False}

B No error, prints 1

C SyntaxError: invalid syntax

D TypeError: duplicated() got an unexpected keyword argument 'keep'

💡 Hint

Check the allowed values for the 'keep' parameter in duplicated().

The 'keep' parameter only accepts 'first', 'last', or False. Using 'maybe' causes a ValueError.

🚀 Application

advanced

Find how many unique duplicate rows exist

How many unique rows appear more than once in this DataFrame?

Pandas

import pandas as pd

df = pd.DataFrame({
    'A': [1, 2, 2, 3, 3, 3, 4],
    'B': ['x', 'y', 'y', 'z', 'z', 'z', 'w']
})

# Count unique rows that have duplicates
counts = df.value_counts()
num_unique_duplicates = (counts > 1).sum()
print(num_unique_duplicates)

A 4

B 2

C 3

D 1

💡 Hint

Use value_counts() to count how many times each row appears, then count how many appear more than once.

Rows (2, 'y') and (3, 'z') appear multiple times. So there are 2 unique rows duplicated. But check carefully: (2, 'y') appears twice, (3, 'z') appears three times. So total unique duplicates are 2.

🧠 Conceptual

expert

Understanding duplicated() with keep=False

What is the output of this code that marks all duplicates including the first occurrence?

Pandas

import pandas as pd

df = pd.DataFrame({
    'A': [2, 2, 2, 3, 3, 3],
    'B': ['y', 'y', 'y', 'z', 'z', 'z']
})

duplicates_all = df.duplicated(keep=False)
print(duplicates_all.sum())

A 0

B 3

C 6

D 4

💡 Hint

The keep=False option marks all duplicates as True, including the first occurrence.

With keep=False, all rows that have duplicates anywhere are marked True. Here, rows with values 2 and 3 appear multiple times, so all those rows are True. That is 6 rows total.

Practice

1.

What does the duplicated() function in pandas do?

easy

A. It counts how many times each value appears in a column.

B. It removes all duplicate rows from the DataFrame.

C. It marks duplicate rows as True and unique rows as False.

D. It sorts the DataFrame by duplicate values.

Solution

1. Step 1: Understand the purpose of duplicated()

   The duplicated() function checks each row and marks it True if it is a duplicate of a previous row, otherwise False.

2. Step 2: Compare with other functions

   Unlike drop_duplicates() which removes duplicates, duplicated() only marks them. value_counts() counts values, not duplicates.

3. Final Answer:

   It marks duplicate rows as True and unique rows as False. -> Option C

4. Quick Check:

   duplicated() = True/False mask [OK]

Hint: duplicated() flags duplicates, does not remove them [OK]

Common Mistakes:

• Thinking duplicated() removes duplicates
• Confusing duplicated() with value_counts()
• Assuming duplicated() sorts data

2.

Which of the following is the correct syntax to count duplicate rows in a pandas DataFrame df?

?

easy

A. df.duplicated().sum()

B. df.duplicate().count()

C. df.value_counts().duplicated()

D. df.count_duplicates()

Solution

1. Step 1: Identify correct function names

   The correct function to find duplicates is duplicated(). There is no duplicate() or count_duplicates() in pandas.

2. Step 2: Count duplicates using sum()

   duplicated() returns a boolean Series. Using sum() counts how many True values (duplicates) exist.

3. Final Answer:

   df.duplicated().sum() -> Option A

4. Quick Check:

   duplicated() + sum() = count duplicates [OK]

Hint: Use duplicated() then sum() to count duplicates [OK]

Common Mistakes:

• Using wrong function names like duplicate()
• Trying to count duplicates with value_counts() directly
• Using count() instead of sum() on boolean Series

3.

Given the DataFrame df below, what is the output of df.duplicated().sum()?

import pandas as pd
df = pd.DataFrame({'A': [1, 2, 2, 3, 1], 'B': ['x', 'y', 'y', 'z', 'x']})
print(df.duplicated().sum())

medium

A. 3

B. 2

C. 1

D. 0

Solution

1. Step 1: Identify duplicate rows

   Rows 0 and 4 are identical (A=1, B='x'), and rows 1 and 2 are identical (A=2, B='y'). The first occurrence is not counted as duplicate.

2. Step 2: Count duplicates marked True

   Row 2 and row 4 are duplicates (second occurrences), so duplicated() marks these two rows True. Summing gives 2.

3. Final Answer:

   2 -> Option B

4. Quick Check:

   Two duplicate rows found = 2 [OK]

Hint: duplicated() counts second+ occurrences only [OK]

Common Mistakes:

• Counting all duplicates including first occurrence
• Confusing duplicated() with value_counts()
• Ignoring that duplicated() marks only later duplicates

4.

What is wrong with this code to count duplicates in DataFrame df?

duplicates = df.duplicated.count()

medium

A. duplicated is a method and needs parentheses before count()

B. count() cannot be used on duplicated() output

C. duplicated() does not exist in pandas

D. sum() should be used instead of count()

Solution

1. Step 1: Check method call syntax

   duplicated is a method and must be called with parentheses: duplicated().

2. Step 2: Understand method chaining

   Without parentheses, df.duplicated.count() tries to access count attribute of the method itself, causing error. Correct is df.duplicated().count(), but this counts all rows, not duplicates.

3. Final Answer:

   duplicated is a method and needs parentheses before count() -> Option A

4. Quick Check:

   Call duplicated() with () before count() [OK]

Hint: Always use parentheses to call duplicated() method [OK]

Common Mistakes:

• Forgetting parentheses on duplicated()
• Using count() instead of sum() to count duplicates
• Assuming duplicated is a property, not a method

5.

You have a DataFrame df with a column 'Category'. You want to find which categories appear more than once and how many times. Which code snippet correctly does this?

?

hard

A. df['Category'].count_duplicates()

B. df.duplicated('Category').sum()

C. df['Category'].duplicated().count()

D. df['Category'].value_counts()[df['Category'].value_counts() > 1]

Solution

1. Step 1: Use value_counts() to count category occurrences

   value_counts() counts how many times each category appears.

2. Step 2: Filter categories with counts greater than 1

   Using boolean indexing [value_counts() > 1] selects only categories appearing more than once.

3. Final Answer:

   df['Category'].value_counts()[df['Category'].value_counts() > 1] -> Option D

4. Quick Check:

   value_counts() + filter > 1 = repeated categories [OK]

Hint: Use value_counts() then filter counts > 1 [OK]

Common Mistakes:

• Using duplicated() on a column returns booleans, not counts
• Trying to call non-existent count_duplicates()
• Using count() on duplicated() output incorrectly