drop_duplicates() for removal in Pandas - Cheat Sheet & Quick Revision

Recall & Review

beginner

What does the drop_duplicates() function do in pandas?

It removes duplicate rows from a DataFrame, keeping only unique rows based on all or selected columns.

beginner

How do you keep the first occurrence of duplicates when using drop_duplicates()?

By default, drop_duplicates() keeps the first occurrence and removes later duplicates. You can also set keep='first' explicitly.

intermediate

What parameter do you use to remove duplicates based on specific columns only?

Use the subset parameter with a list of column names to consider only those columns when identifying duplicates.

intermediate

How can you remove duplicates and modify the original DataFrame directly?

Set the parameter inplace=True in drop_duplicates() to remove duplicates without creating a new DataFrame.

advanced

What happens if you set keep=False in drop_duplicates()?

All duplicates are removed, including the first occurrences, so only rows that are unique remain.

What is the default behavior of drop_duplicates() in pandas?

A Remove all duplicates including first occurrences

B Keep the first occurrence of duplicates

C Keep the last occurrence of duplicates

D Do nothing

By default, drop_duplicates() keeps the first occurrence and removes later duplicates.

Which parameter lets you specify columns to check for duplicates?

A columns

B axis

C filter

D subset

The subset parameter takes a list of columns to consider when identifying duplicates.

How do you remove duplicates and update the original DataFrame without creating a new one?

A Set <code>inplace=True</code>

B Set <code>keep=False</code>

C Use <code>drop()</code> instead

D Use <code>reset_index()</code>

Using inplace=True modifies the original DataFrame directly.

What does keep=False do in drop_duplicates()?

A Removes all duplicates including first occurrences

B Keeps the last duplicate

C Keeps all duplicates

D Keeps the first duplicate

Setting keep=False removes all duplicates, leaving only unique rows.

If you want to remove duplicates based on columns 'A' and 'B' only, which code is correct?

A df.drop_duplicates(keep=['A', 'B'])

B df.drop_duplicates(columns=['A', 'B'])

C df.drop_duplicates(subset=['A', 'B'])

D df.drop_duplicates(axis=['A', 'B'])

The subset parameter specifies columns to check for duplicates.

Explain how drop_duplicates() works and how you can control which duplicates to keep or remove.

Think about how to keep or remove duplicates and which columns to consider.

Describe a real-life example where removing duplicates from data is important and how drop_duplicates() helps.

Imagine cleaning a list of names or transactions.

Practice

1. What does the drop_duplicates() function do in pandas?

easy

A. Removes repeated rows from a DataFrame

B. Adds new rows to a DataFrame

C. Sorts the DataFrame by column values

D. Changes the data type of columns

Solution

1. Step 1: Understand the purpose of drop_duplicates()

   This function is designed to remove rows that appear more than once in a DataFrame.

2. Step 2: Compare options with function behavior

   Only Removes repeated rows from a DataFrame correctly describes removing repeated rows. Other options describe different functions.

3. Final Answer:

   Removes repeated rows from a DataFrame -> Option A

4. Quick Check:

   drop_duplicates() removes repeats [OK]

Hint: Remember: drop_duplicates() cleans repeated rows fast [OK]

Common Mistakes:

• Confusing drop_duplicates() with sorting functions
• Thinking it adds or modifies data instead of removing duplicates
• Assuming it changes data types

2. Which of the following is the correct syntax to remove duplicate rows from a DataFrame df?

easy

A. df.drop_duplicate_rows()

B. df.remove_duplicates()

C. df.delete_duplicates()

D. df.drop_duplicates()

Solution

1. Step 1: Recall the exact pandas method name

   The correct method to remove duplicates is drop_duplicates().

2. Step 2: Check each option for correct syntax

   Only df.drop_duplicates() uses the correct method name and syntax. Others are invalid pandas methods.

3. Final Answer:

   df.drop_duplicates() -> Option D

4. Quick Check:

   Correct method name is drop_duplicates() [OK]

Hint: Use exact method name: drop_duplicates() [OK]

Common Mistakes:

• Using incorrect method names like remove_duplicates()
• Adding extra words like delete or rows
• Confusing with other DataFrame methods

3. Given the DataFrame df below:

import pandas as pd
data = {'A': [1, 2, 2, 3], 'B': ['x', 'y', 'y', 'z']}
df = pd.DataFrame(data)
print(df.drop_duplicates())

What will be the output?

medium

A. Rows with index 1, 2, and 3

B. Rows with index 0, 1, and 3

C. Rows with index 0, 2, and 3

D. All rows remain unchanged

Solution

1. Step 1: Identify duplicate rows in the DataFrame

   Rows 2 and 3 have the same values (A=2, B='y'). So one will be removed.

2. Step 2: Understand default behavior of drop_duplicates()

   By default, it keeps the first occurrence and removes later duplicates. So row 1 stays, row 2 is removed.

3. Final Answer:

   Rows with index 0, 1, and 3 -> Option B

4. Quick Check:

   First duplicates kept, later removed [OK]

Hint: drop_duplicates() keeps first by default [OK]

Common Mistakes:

• Assuming last duplicate is kept instead of first
• Thinking all duplicates are removed including first
• Confusing row indices after removal

4. What is wrong with this code snippet to remove duplicates from DataFrame df?

df.drop_duplicates(inplace=True)
print(df.drop_duplicates())

medium

A. Calling drop_duplicates() twice is redundant

B. Using inplace=True and then printing drop_duplicates() returns original data

C. drop_duplicates() does not have inplace parameter

D. print statement syntax is incorrect

Solution

1. Step 1: Understand inplace=True effect

   Using inplace=True modifies df directly by removing duplicates.

2. Step 2: Analyze print statement

   After the first call, df is already deduplicated. The second call to df.drop_duplicates() is redundant as it returns the same DataFrame since there are no duplicates left.

3. Final Answer:

   Calling drop_duplicates() twice is redundant -> Option A

4. Quick Check:

   Second call redundant after inplace [OK]

Hint: inplace=True changes df; extra drop_duplicates() returns copy [OK]

Common Mistakes:

• Thinking drop_duplicates() lacks inplace parameter
• Assuming print statement is wrong syntax
• Believing calling twice causes error

5. You have a DataFrame df with columns ['Name', 'Age', 'City']. You want to remove duplicates only based on Name and City, keeping the last occurrence. Which code achieves this?

hard

A. df.drop_duplicates(columns=['Name', 'City'], keep='last')

B. df.drop_duplicates(subset=['Name', 'City'], keep='first')

C. df.drop_duplicates(subset=['Name', 'City'], keep='last')

D. df.drop_duplicates(subset=['Age'], keep='last')

Solution

1. Step 1: Use subset parameter to specify columns

   To check duplicates only on 'Name' and 'City', use subset=['Name', 'City'].

2. Step 2: Set keep='last' to keep last occurrence

   keep='last' keeps the last duplicate row, removing earlier ones.

3. Final Answer:

   df.drop_duplicates(subset=['Name', 'City'], keep='last') -> Option C

4. Quick Check:

   subset selects columns; keep='last' keeps last duplicate [OK]

Hint: Use subset for columns, keep='last' to keep last duplicate [OK]

Common Mistakes:

• Using 'columns' instead of 'subset' parameter
• Forgetting to set keep='last' to keep last occurrence
• Selecting wrong columns for subset