duplicated() for finding duplicates in Pandas - Cheat Sheet & Quick Revision

Recall & Review

beginner

What does the duplicated() function in pandas do?

It returns a Boolean Series indicating whether each row is a duplicate of a previous row in the DataFrame.

beginner

How can you use duplicated() to find all duplicate rows except the first occurrence?

By default, duplicated() marks duplicates as True except for the first occurrence, which is False.

intermediate

What parameter of duplicated() controls which duplicates are marked as True?

The keep parameter controls this. It can be 'first' (default), 'last', or False to mark all duplicates as True.

intermediate

How do you find duplicates based on specific columns using duplicated()?

Use the subset parameter to specify columns to check for duplicates instead of the entire row.

intermediate

What is the difference between duplicated() and drop_duplicates()?

duplicated() returns a Boolean mask showing duplicates, while drop_duplicates() returns a DataFrame with duplicates removed.

What does df.duplicated() return?

A A DataFrame with duplicates removed

B The count of duplicate rows

C A Boolean Series marking duplicate rows except the first occurrence

D The unique rows only

duplicated() returns a Boolean Series marking duplicates as True except the first occurrence.

Which keep parameter value marks all duplicates as True in duplicated()?

A 'last'

B False

C 'first'

D None

Setting keep=False marks all duplicates as True.

How do you check duplicates based on only some columns?

A Use <code>unique()</code> on those columns

B Use <code>drop_duplicates()</code> without parameters

C Use <code>groupby()</code> only

D Use the <code>subset</code> parameter in <code>duplicated()</code>

The subset parameter lets you specify columns to check for duplicates.

What does df.duplicated(keep='last') do?

A Marks duplicates as True except the last occurrence

B Marks duplicates as True except the first occurrence

C Marks all duplicates as True

D Removes duplicates from DataFrame

With keep='last', all duplicates except the last occurrence are marked True.

Which function returns a DataFrame with duplicates removed?

A <code>drop_duplicates()</code>

B <code>duplicated()</code>

C <code>unique()</code>

D <code>isnull()</code>

drop_duplicates() returns a DataFrame with duplicates removed.

Explain how to use duplicated() to find duplicate rows in a DataFrame and how to customize which duplicates are marked.

Think about which rows are marked True and how to change that.

Describe the difference between duplicated() and drop_duplicates() and when you might use each.

Consider output type and purpose.

Practice

1. What does the duplicated() function in pandas do?

easy

A. It identifies duplicate rows in a DataFrame.

B. It removes duplicate rows from a DataFrame.

C. It sorts the DataFrame by duplicate values.

D. It counts the number of unique rows in a DataFrame.

Solution

1. Step 1: Understand the purpose of duplicated()

   The duplicated() function checks each row and marks it as True if it is a duplicate of a previous row.

2. Step 2: Differentiate from other functions

   Unlike functions that remove duplicates or count unique rows, duplicated() only flags duplicates without changing the DataFrame.

3. Final Answer:

   It identifies duplicate rows in a DataFrame. -> Option A

4. Quick Check:

   duplicated() = identifies duplicates [OK]

Hint: Remember: duplicated() flags duplicates, doesn't remove them [OK]

Common Mistakes:

• Confusing duplicated() with drop_duplicates()
• Thinking duplicated() sorts data
• Assuming duplicated() counts unique rows

2. Which of the following is the correct syntax to find duplicates in a pandas DataFrame df?

easy

A. df.duplicated()

B. df.duplicates()

C. df.find_duplicates()

D. df.duplicate()

Solution

1. Step 1: Recall the exact function name

   The correct pandas function to find duplicate rows is duplicated(), not any other variation.

2. Step 2: Verify syntax correctness

   The function is called on the DataFrame object as df.duplicated() with parentheses.

3. Final Answer:

   df.duplicated() -> Option A

4. Quick Check:

   Correct function name = df.duplicated() [OK]

Hint: Use exact function name: duplicated() with parentheses [OK]

Common Mistakes:

• Misspelling the function name
• Omitting parentheses after duplicated
• Using non-existent functions like duplicates()

3. Given the DataFrame df below, what is the output of df.duplicated()?

import pandas as pd
data = {'A': [1, 2, 2, 3], 'B': ['x', 'y', 'y', 'z']}
df = pd.DataFrame(data)
print(df.duplicated())

medium

A. [True, False, True, False]

B. [False, False, True, False]

C. [False, True, False, True]

D. [False, False, False, True]

Solution

1. Step 1: Identify duplicate rows in the DataFrame

   The rows are: 0: (1, 'x'), 1: (2, 'y'), 2: (2, 'y'), 3: (3, 'z'). Row 2 is a duplicate of row 1.

2. Step 2: Understand duplicated() output

   duplicated() returns False for the first occurrence and True for duplicates. So, rows 0,1,3 are False; row 2 is True.

3. Final Answer:

   [False, False, True, False] -> Option B

4. Quick Check:

   First duplicate row marked True = [False, False, True, False] [OK]

Hint: First occurrence is False, duplicates after are True [OK]

Common Mistakes:

• Marking first occurrence as True
• Confusing row indices with values
• Assuming all duplicates are True regardless of position

4. What is wrong with this code snippet to find duplicates in column 'A' of DataFrame df?

duplicates = df.duplicated['A']
print(duplicates)

medium

A. duplicated() cannot check specific columns

B. Missing import statement for pandas

C. The print statement is incorrect

D. Using square brackets instead of parentheses to call duplicated()

Solution

1. Step 1: Check function call syntax

   The code uses square brackets [] instead of parentheses () to call the duplicated() method, which is incorrect syntax.

2. Step 2: Correct way to specify columns

   To check duplicates based on column 'A', the correct syntax is df.duplicated(subset=['A']) with parentheses.

3. Final Answer:

   Using square brackets instead of parentheses to call duplicated() -> Option D

4. Quick Check:

   Call methods with () not [] [OK]

Hint: Call duplicated() with parentheses, not square brackets [OK]

Common Mistakes:

• Using [] instead of () to call functions
• Trying to access duplicated as an attribute
• Not using subset parameter for specific columns

5. You have a DataFrame df with columns 'Name', 'Age', and 'City'. You want to find duplicates only based on 'Name' and 'City', ignoring 'Age'. Which code correctly finds these duplicates?

hard

A. df.duplicated(['Name', 'City'], keep=True)

B. df.duplicated(columns=['Name', 'City'])

C. df.duplicated(subset=['Name', 'City'])

D. df.duplicated('Name', 'City')

Solution

1. Step 1: Use subset parameter to specify columns

   To check duplicates based on specific columns, use the subset parameter with a list of column names.

2. Step 2: Validate correct syntax

   The correct syntax is df.duplicated(subset=['Name', 'City']). df.duplicated(columns=['Name', 'City']) uses a wrong parameter name (columns), and options A and C use invalid syntax.

3. Final Answer:

   df.duplicated(subset=['Name', 'City']) -> Option C

4. Quick Check:

   Use subset=['col1', 'col2'] to check specific columns [OK]

Hint: Use subset parameter to check duplicates on chosen columns [OK]

Common Mistakes:

• Using wrong parameter name like columns instead of subset
• Passing list without parameter name
• Confusing keep parameter with subset