drop_duplicates() for removal in Pandas - Step-by-Step Execution

Concept Flow - drop_duplicates() for removal

Start with DataFrame

↓

Call drop_duplicates()

↓

Check each row for duplicates

↓

Keep first occurrence, remove others

↓

Return new DataFrame without duplicates

↓

End

The function scans the DataFrame rows, keeps the first occurrence of duplicates, removes the rest, and returns a new DataFrame.

Execution Sample

Pandas

import pandas as pd

df = pd.DataFrame({
    'A': [1, 2, 2, 3],
    'B': ['x', 'y', 'y', 'z']
})

result = df.drop_duplicates()

This code creates a DataFrame with duplicate rows and removes duplicates using drop_duplicates().

Execution Table

Step	Row Index	Row Data	Is Duplicate?	Action	Resulting DataFrame Rows
1	0	{'A': 1, 'B': 'x'}	No	Keep	[0]
2	1	{'A': 2, 'B': 'y'}	No	Keep	[0, 1]
3	2	{'A': 2, 'B': 'y'}	Yes (duplicate of index 1)	Remove	[0, 1]
4	3	{'A': 3, 'B': 'z'}	No	Keep	[0, 1, 3]
5	-	-	-	-	Duplicates removed, final rows: [0, 1, 3]

💡 All rows checked; duplicates removed; final DataFrame has rows with indices 0, 1, and 3.

Variable Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4	Final
df	Original DataFrame with 4 rows	Same	Same	Same	Same	Same
result	Undefined	Undefined	Undefined	Undefined	Undefined	DataFrame with rows 0,1,3

Key Moments - 2 Insights

Why does drop_duplicates() keep the first occurrence and remove later ones?

drop_duplicates() by default keeps the first occurrence to preserve the original order and remove repeated rows seen later, as shown in execution_table rows 2 and 3.

Does drop_duplicates() modify the original DataFrame?

No, drop_duplicates() returns a new DataFrame without duplicates. The original DataFrame 'df' remains unchanged, as shown in variable_tracker.

Visual Quiz - 3 Questions

Look at the execution table, which row is identified as a duplicate and removed?

A Row index 3

B Row index 1

C Row index 2

D Row index 0

💡 Hint

Check the 'Is Duplicate?' column in execution_table row 3.

At which step does the resulting DataFrame first exclude a duplicate row?

A Step 2

B Step 3

C Step 4

D Step 1

💡 Hint

Look at the 'Action' column in execution_table rows to see when 'Remove' happens.

If drop_duplicates() was called with keep='last', which row would be removed instead?

A Row index 1

B Row index 0

C Row index 2

D Row index 3

💡 Hint

With keep='last', the first duplicate is removed, so check which row is the first duplicate in execution_table.

Concept Snapshot

drop_duplicates() removes duplicate rows from a DataFrame.
By default, it keeps the first occurrence and removes later duplicates.
It returns a new DataFrame; original stays unchanged.
You can specify columns or keep='last' to change behavior.
Useful to clean repeated data easily.

Full Transcript

We start with a DataFrame containing some duplicate rows. When we call drop_duplicates(), it checks each row in order. If a row is the first time it appears, it keeps it. If it finds a row that matches a previous one, it marks it as duplicate and removes it. The function returns a new DataFrame without those duplicates. The original DataFrame remains unchanged. This process helps clean data by removing repeated rows while keeping the first instance. You can also change which duplicates to keep by parameters, but by default, it keeps the first. This is shown step-by-step in the execution table where row index 2 is removed because it duplicates row index 1.

Practice

1. What does the drop_duplicates() function do in pandas?

easy

A. Removes repeated rows from a DataFrame

B. Adds new rows to a DataFrame

C. Sorts the DataFrame by column values

D. Changes the data type of columns

Solution

1. Step 1: Understand the purpose of drop_duplicates()

   This function is designed to remove rows that appear more than once in a DataFrame.

2. Step 2: Compare options with function behavior

   Only Removes repeated rows from a DataFrame correctly describes removing repeated rows. Other options describe different functions.

3. Final Answer:

   Removes repeated rows from a DataFrame -> Option A

4. Quick Check:

   drop_duplicates() removes repeats [OK]

Hint: Remember: drop_duplicates() cleans repeated rows fast [OK]

Common Mistakes:

• Confusing drop_duplicates() with sorting functions
• Thinking it adds or modifies data instead of removing duplicates
• Assuming it changes data types

2. Which of the following is the correct syntax to remove duplicate rows from a DataFrame df?

easy

A. df.drop_duplicate_rows()

B. df.remove_duplicates()

C. df.delete_duplicates()

D. df.drop_duplicates()

Solution

1. Step 1: Recall the exact pandas method name

   The correct method to remove duplicates is drop_duplicates().

2. Step 2: Check each option for correct syntax

   Only df.drop_duplicates() uses the correct method name and syntax. Others are invalid pandas methods.

3. Final Answer:

   df.drop_duplicates() -> Option D

4. Quick Check:

   Correct method name is drop_duplicates() [OK]

Hint: Use exact method name: drop_duplicates() [OK]

Common Mistakes:

• Using incorrect method names like remove_duplicates()
• Adding extra words like delete or rows
• Confusing with other DataFrame methods

3. Given the DataFrame df below:

import pandas as pd
data = {'A': [1, 2, 2, 3], 'B': ['x', 'y', 'y', 'z']}
df = pd.DataFrame(data)
print(df.drop_duplicates())

What will be the output?

medium

A. Rows with index 1, 2, and 3

B. Rows with index 0, 1, and 3

C. Rows with index 0, 2, and 3

D. All rows remain unchanged

Solution

1. Step 1: Identify duplicate rows in the DataFrame

   Rows 2 and 3 have the same values (A=2, B='y'). So one will be removed.

2. Step 2: Understand default behavior of drop_duplicates()

   By default, it keeps the first occurrence and removes later duplicates. So row 1 stays, row 2 is removed.

3. Final Answer:

   Rows with index 0, 1, and 3 -> Option B

4. Quick Check:

   First duplicates kept, later removed [OK]

Hint: drop_duplicates() keeps first by default [OK]

Common Mistakes:

• Assuming last duplicate is kept instead of first
• Thinking all duplicates are removed including first
• Confusing row indices after removal

4. What is wrong with this code snippet to remove duplicates from DataFrame df?

df.drop_duplicates(inplace=True)
print(df.drop_duplicates())

medium

A. Calling drop_duplicates() twice is redundant

B. Using inplace=True and then printing drop_duplicates() returns original data

C. drop_duplicates() does not have inplace parameter

D. print statement syntax is incorrect

Solution

1. Step 1: Understand inplace=True effect

   Using inplace=True modifies df directly by removing duplicates.

2. Step 2: Analyze print statement

   After the first call, df is already deduplicated. The second call to df.drop_duplicates() is redundant as it returns the same DataFrame since there are no duplicates left.

3. Final Answer:

   Calling drop_duplicates() twice is redundant -> Option A

4. Quick Check:

   Second call redundant after inplace [OK]

Hint: inplace=True changes df; extra drop_duplicates() returns copy [OK]

Common Mistakes:

• Thinking drop_duplicates() lacks inplace parameter
• Assuming print statement is wrong syntax
• Believing calling twice causes error

5. You have a DataFrame df with columns ['Name', 'Age', 'City']. You want to remove duplicates only based on Name and City, keeping the last occurrence. Which code achieves this?

hard

A. df.drop_duplicates(columns=['Name', 'City'], keep='last')

B. df.drop_duplicates(subset=['Name', 'City'], keep='first')

C. df.drop_duplicates(subset=['Name', 'City'], keep='last')

D. df.drop_duplicates(subset=['Age'], keep='last')

Solution

1. Step 1: Use subset parameter to specify columns

   To check duplicates only on 'Name' and 'City', use subset=['Name', 'City'].

2. Step 2: Set keep='last' to keep last occurrence

   keep='last' keeps the last duplicate row, removing earlier ones.

3. Final Answer:

   df.drop_duplicates(subset=['Name', 'City'], keep='last') -> Option C

4. Quick Check:

   subset selects columns; keep='last' keeps last duplicate [OK]

Hint: Use subset for columns, keep='last' to keep last duplicate [OK]

Common Mistakes:

• Using 'columns' instead of 'subset' parameter
• Forgetting to set keep='last' to keep last occurrence
• Selecting wrong columns for subset