Duplicates on specific columns in Pandas - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of drop_duplicates on specific columns

What is the output DataFrame after running the code below?

Pandas

import pandas as pd

data = {'A': [1, 2, 2, 3, 4], 'B': [5, 6, 6, 7, 8], 'C': [9, 10, 11, 12, 13]}
df = pd.DataFrame(data)
result = df.drop_duplicates(subset=['A', 'B'])
print(result)

A {'A': [1, 2, 2, 3, 4], 'B': [5, 6, 6, 7, 8], 'C': [9, 10, 11, 12, 13]}

B {'A': [1, 2, 3, 4], 'B': [5, 6, 7, 8], 'C': [9, 10, 12, 13]}

C {'A': [1, 2, 3, 4], 'B': [5, 6, 7, 8], 'C': [9, 11, 12, 13]}

D {'A': [1, 2, 2, 3], 'B': [5, 6, 6, 7], 'C': [9, 10, 11, 12]}

💡 Hint

drop_duplicates keeps the first occurrence of each duplicate based on the subset columns.

The method drop_duplicates with subset=['A', 'B'] removes rows where the combination of columns A and B repeats. The first occurrence is kept, so the second row with A=2 and B=6 is kept, but the third row with the same values is dropped.

❓ data_output

intermediate

Count of duplicate rows on specific columns

What is the number of duplicate rows based on columns 'X' and 'Y' in the DataFrame?

Pandas

import pandas as pd

data = {'X': [1, 2, 2, 3, 3, 3], 'Y': ['a', 'b', 'b', 'c', 'c', 'd'], 'Z': [10, 20, 20, 30, 30, 40]}
df = pd.DataFrame(data)
duplicates = df.duplicated(subset=['X', 'Y'])
count = duplicates.sum()
print(count)

A 1

B 3

C 4

D 2

💡 Hint

duplicated() marks all but the first occurrence as duplicates.

Rows with (X=2, Y='b') and (X=3, Y='c') each have duplicates. The second occurrence of each is marked true, so total duplicates are 2.

🔧 Debug

advanced

Identify the error in duplicate filtering code

What error will the following code produce?

Pandas

import pandas as pd

data = {'A': [1, 2, 2], 'B': [3, 4, 4]}
df = pd.DataFrame(data)
filtered = df[df.duplicated(columns=['A'])]
print(filtered)

A No error, prints duplicate rows based on column 'A'

B KeyError: 'columns'

C TypeError: duplicated() got an unexpected keyword argument 'columns'

D ValueError: subset columns not found

💡 Hint

Check the parameter name for specifying columns in duplicated().

The duplicated() method uses 'subset' as the parameter name for columns, not 'columns'. Using 'columns' causes a TypeError.

🚀 Application

advanced

Filter DataFrame to keep only duplicates on columns 'M' and 'N'

Which code snippet correctly filters the DataFrame to keep only rows that have duplicates based on columns 'M' and 'N'?

Pandas

import pandas as pd

data = {'M': [1, 2, 2, 3, 3, 3], 'N': ['x', 'y', 'y', 'z', 'z', 'w'], 'O': [5, 6, 7, 8, 9, 10]}
df = pd.DataFrame(data)

A df[df.duplicated(subset=['M', 'N'], keep=False)]

B df[df.duplicated(subset=['M', 'N'])]

C df.drop_duplicates(subset=['M', 'N'], keep=False)

D df[df.duplicated(subset=['M', 'N'], keep='first')]

💡 Hint

Use keep=False to mark all duplicates, not just later ones.

duplicated(keep=False) marks all duplicates true, so filtering with it keeps all duplicate rows. Other options either keep only later duplicates or remove duplicates.

🧠 Conceptual

expert

Effect of keep parameter in drop_duplicates on specific columns

Consider a DataFrame with duplicate rows based on columns 'P' and 'Q'. What is the difference in output between drop_duplicates(subset=['P', 'Q'], keep='first') and drop_duplicates(subset=['P', 'Q'], keep='last')?

A keep='first' keeps the first occurrence of each duplicate, keep='last' keeps the last occurrence; other duplicates are dropped accordingly.

B keep='first' keeps duplicates with the smallest index, keep='last' raises an error.

C keep='first' and keep='last' produce identical outputs.

D keep='first' drops all duplicates, keep='last' keeps all duplicates.

💡 Hint

Think about which duplicate row is kept when using 'first' or 'last'.

The keep parameter controls which duplicate row is kept: 'first' keeps the earliest row, 'last' keeps the latest row. The rest are dropped.

Practice

1. What does the subset parameter do in the duplicated() method in pandas?

easy

A. It specifies which columns to check for duplicates.

B. It removes duplicates from the DataFrame.

C. It sorts the DataFrame before checking duplicates.

D. It changes the data type of columns before checking duplicates.

Solution

1. Step 1: Understand the purpose of duplicated()

   The duplicated() method identifies duplicate rows in a DataFrame.

2. Step 2: Role of subset parameter

   The subset parameter limits the duplicate check to specific columns instead of all columns.

3. Final Answer:

   It specifies which columns to check for duplicates. -> Option A

4. Quick Check:

   subset = columns to check duplicates [OK]

Hint: Use subset to focus duplicate check on chosen columns [OK]

Common Mistakes:

• Thinking subset removes duplicates
• Confusing subset with sorting
• Assuming subset changes data types

2. Which of the following is the correct syntax to find duplicates based on columns 'A' and 'B' in a pandas DataFrame df?

easy

A. df.duplicated(subset=['A', 'B'])

B. df.duplicates(columns=['A', 'B'])

C. df.duplicated(columns=['A', 'B'])

D. df.find_duplicates(['A', 'B'])

Solution

1. Step 1: Recall pandas method for duplicates

   The correct method is duplicated() to find duplicate rows.

2. Step 2: Correct use of subset parameter

   The subset parameter takes a list of column names to check duplicates on.

3. Final Answer:

   df.duplicated(subset=['A', 'B']) -> Option A

4. Quick Check:

   Correct method and subset syntax [OK]

Hint: Use duplicated(subset=[...]) to specify columns [OK]

Common Mistakes:

• Using non-existent methods like duplicates()
• Passing columns instead of subset
• Wrong method names like find_duplicates()

3. Given the DataFrame:

import pandas as pd
data = {'A': [1, 2, 1, 3], 'B': [4, 5, 4, 6], 'C': [7, 8, 9, 10]}
df = pd.DataFrame(data)
print(df.duplicated(subset=['A', 'B']))

What is the output?

medium

A. [False, True, False, False]

B. [True, False, True, False]

C. [False, False, True, False]

D. [False, False, False, True]

Solution

1. Step 1: Understand the DataFrame and subset

   The DataFrame has rows with columns A and B: (1,4), (2,5), (1,4), (3,6).

2. Step 2: Identify duplicates on columns 'A' and 'B'

   The third row (index 2) duplicates the first row (index 0) on columns A and B.

3. Final Answer:

   [False, False, True, False] -> Option C

4. Quick Check:

   Only third row is duplicate on A and B [OK]

Hint: Check repeated (A,B) pairs for True in duplicated() output [OK]

Common Mistakes:

• Assuming duplicates on all columns
• Marking first occurrence as duplicate
• Confusing index positions

4. You run this code:

df.duplicated(subset=['A', 'B'], keep='last')

but get unexpected results. What is a common mistake causing this?

medium

A. Forgetting to pass a list to subset causes an error.

B. Using keep='last' marks the last occurrence as False, others as True.

C. Using keep='last' marks the first duplicates as True, not the last.

D. The duplicated() method does not accept keep parameter.

Solution

1. Step 1: Understand keep parameter behavior

   keep='last' marks all duplicates except the last occurrence as True.

2. Step 2: Identify common confusion

   Many expect it to mark the last duplicate as True, but it actually keeps the last as False (not duplicate).

3. Final Answer:

   Using keep='last' marks the last occurrence as False, others as True. -> Option B

4. Quick Check:

   keep='last' keeps last duplicate False [OK]

Hint: Remember keep='last' keeps last duplicate False, others True [OK]

Common Mistakes:

• Thinking keep='last' marks last duplicate True
• Passing subset as string instead of list
• Believing duplicated() lacks keep parameter

5. You have a DataFrame df with columns 'Name', 'Age', 'City'. You want to remove rows where 'Name' and 'City' are duplicates but keep the first occurrence. Which code correctly does this?

hard

A. df[~df.duplicated(subset=['Name', 'City'], keep='first')]

B. df = df.duplicated(subset=['Name', 'City'], keep='first')

C. df = df.drop_duplicates(columns=['Name', 'City'], keep='last')

D. df = df.drop_duplicates(subset=['Name', 'City'], keep='first')

Solution

1. Step 1: Identify method to remove duplicates

   Use drop_duplicates() to remove duplicate rows.

2. Step 2: Use subset and keep parameters correctly

   Pass subset=['Name', 'City'] and keep='first' to keep first occurrence and remove others.

3. Step 3: Check options

   df = df.drop_duplicates(subset=['Name', 'City'], keep='first') uses correct method and parameters. df[~df.duplicated(subset=['Name', 'City'], keep='first')] uses duplicated() with negation but does not assign back to df. df = df.drop_duplicates(columns=['Name', 'City'], keep='last') uses wrong parameter name. df = df.duplicated(subset=['Name', 'City'], keep='first') returns boolean Series, not DataFrame.

4. Final Answer:

   df = df.drop_duplicates(subset=['Name', 'City'], keep='first') -> Option D

5. Quick Check:

   drop_duplicates with subset and keep='first' removes duplicates [OK]

Hint: Use drop_duplicates(subset=[...], keep='first') to remove duplicates [OK]

Common Mistakes:

• Using duplicated() instead of drop_duplicates() to remove rows
• Passing columns instead of subset
• Not assigning result back to df