Keeping first vs last vs none in Pandas - Visual Side-by-Side Comparison

Concept Flow - Keeping first vs last vs none

Start with DataFrame

↓

Identify duplicates

↓

Choose keep option

↓

Keep first duplicate

↓

Return cleaned DataFrame

This flow shows how pandas identifies duplicates and then keeps either the first, last, or no duplicates based on the chosen option.

Execution Sample

Pandas

import pandas as pd

df = pd.DataFrame({'A': [1, 2, 2, 3, 3, 3], 'B': ['x', 'y', 'y', 'z', 'z', 'z']})
df_first = df.drop_duplicates(keep='first')
df_last = df.drop_duplicates(keep='last')
df_none = df.drop_duplicates(keep=False)

This code creates a DataFrame with duplicates and shows how to keep first, last, or no duplicates.

Execution Table

Step	DataFrame State	Action	Resulting DataFrame Rows
1	[1,x], [2,y], [2,y], [3,z], [3,z], [3,z]	Identify duplicates	Duplicates found at rows 2,3 (2,y) and rows 4,5,6 (3,z)
2	[1,x], [2,y], [2,y], [3,z], [3,z], [3,z]	drop_duplicates(keep='first')	[1,x], [2,y], [3,z]
3	[1,x], [2,y], [2,y], [3,z], [3,z], [3,z]	drop_duplicates(keep='last')	[1,x], [2,y], [3,z]
4	[1,x], [2,y], [2,y], [3,z], [3,z], [3,z]	drop_duplicates(keep=False)	[1,x]
5		End of operations	No more duplicates to process

💡 All duplicates handled according to keep option; process ends.

Variable Tracker

Variable	Start	After Step 2	After Step 3	After Step 4	Final
df	[1,x], [2,y], [2,y], [3,z], [3,z], [3,z]	[1,x], [2,y], [2,y], [3,z], [3,z], [3,z]	[1,x], [2,y], [2,y], [3,z], [3,z], [3,z]	[1,x], [2,y], [2,y], [3,z], [3,z], [3,z]	[1,x], [2,y], [2,y], [3,z], [3,z], [3,z]
df_first	N/A	[1,x], [2,y], [3,z]	[1,x], [2,y], [3,z]	[1,x], [2,y], [3,z]	[1,x], [2,y], [3,z]
df_last	N/A	N/A	[1,x], [2,y], [3,z]	[1,x], [2,y], [3,z]	[1,x], [2,y], [3,z]
df_none	N/A	N/A	N/A	[1,x]	[1,x]

Key Moments - 3 Insights

Why does drop_duplicates(keep='first') keep the first occurrence and remove later ones?

Because pandas scans rows top to bottom and marks duplicates after the first occurrence for removal, as shown in execution_table step 2.

What happens when keep=False is used?

All rows that have duplicates anywhere are removed, leaving only unique rows, as shown in execution_table step 4 where only [1,x] remains.

Why do drop_duplicates(keep='first') and drop_duplicates(keep='last') sometimes return the same rows?

If duplicates are identical, keeping first or last results in one row per duplicate group, so the output rows look the same, as seen in steps 2 and 3.

Visual Quiz - 3 Questions

Look at the execution_table at step 4. What rows remain after drop_duplicates(keep=False)?

A [2,y], [3,z]

B [1,x]

C [1,x], [2,y], [3,z]

D All rows remain

💡 Hint

Check the Resulting DataFrame Rows column at step 4 in execution_table.

At which step does drop_duplicates(keep='last') produce its result?

A Step 3

B Step 4

C Step 2

D Step 5

💡 Hint

Look at the Action column in execution_table for drop_duplicates(keep='last').

If the DataFrame had no duplicates, what would drop_duplicates(keep='first') return?

A An empty DataFrame

B Only the first row

C The original DataFrame unchanged

D Only the last row

💡 Hint

Consider what happens when no duplicates are found in the initial DataFrame state.

Concept Snapshot

pandas drop_duplicates method removes duplicate rows.
keep='first' keeps the first occurrence.
keep='last' keeps the last occurrence.
keep=False removes all duplicates.
Useful to clean data by controlling which duplicates to keep.

Full Transcript

This lesson shows how pandas drop_duplicates works with keep options: first, last, and none. We start with a DataFrame containing duplicate rows. The method identifies duplicates and then removes them based on the keep parameter. If keep='first', it keeps the first occurrence and removes later duplicates. If keep='last', it keeps the last occurrence and removes earlier duplicates. If keep=False, it removes all rows that have duplicates, leaving only unique rows. The execution table traces these steps with the DataFrame states and results. Variable tracking shows how the DataFrames change after each operation. Key moments clarify common confusions about how duplicates are handled. The visual quiz tests understanding by asking about the resulting rows after each operation. This helps beginners see exactly how pandas manages duplicates step-by-step.

Practice

1.

In pandas, what does keep='first' do when removing duplicates?

easy

A. Keeps the first occurrence of each duplicate and removes the rest

B. Keeps the last occurrence of each duplicate and removes the rest

C. Removes all duplicates, keeping only unique rows

D. Keeps all duplicates without removing any

Solution

1. Step 1: Understand the meaning of keep='first'

   This option keeps the first row of each set of duplicates and removes the later ones.

2. Step 2: Compare with other options

   Unlike keep='last' which keeps the last occurrence, keep='first' keeps the earliest one.

3. Final Answer:

   Keeps the first occurrence of each duplicate and removes the rest -> Option A

4. Quick Check:

   keep='first' = Keep first duplicate [OK]

Hint: Remember: 'first' keeps earliest duplicate row [OK]

Common Mistakes:

• Confusing keep='first' with keep='last'
• Thinking keep='first' removes all duplicates
• Assuming keep='first' keeps all duplicates

2.

Which of the following is the correct syntax to remove duplicates in a pandas DataFrame df keeping only the last occurrence?

df._____('column_name', keep='last')

easy

A. drop_dupes

B. remove_duplicates

C. drop_duplicates

D. delete_duplicates

Solution

1. Step 1: Identify the pandas method for removing duplicates

   The correct method is drop_duplicates to remove duplicate rows.

2. Step 2: Confirm syntax for keeping last occurrence

   Using keep='last' inside drop_duplicates keeps the last duplicate row.

3. Final Answer:

   drop_duplicates -> Option C

4. Quick Check:

   Method to remove duplicates = drop_duplicates [OK]

Hint: Use df.drop_duplicates() to remove duplicates [OK]

Common Mistakes:

• Using non-existent methods like remove_duplicates
• Confusing method names with similar words
• Forgetting to specify keep='last'

3.

What will be the output of the following code?

import pandas as pd

data = {'A': [1, 2, 2, 3, 3, 3], 'B': ['x', 'y', 'y', 'z', 'z', 'z']}
df = pd.DataFrame(data)
result = df.drop_duplicates(keep=False)
print(result)

medium

A. Rows with unique values only, no duplicates

B. All rows except the first occurrence of duplicates

C. Rows with A=1 B='x', A=2 B='y', A=3 B='z' only

D. Empty DataFrame

Solution

1. Step 1: Understand keep=False behavior

   Using keep=False removes all duplicates, keeping only rows that appear once.

2. Step 2: Identify unique rows in the DataFrame

   In the data, only the row with A=1 and B='x' is unique; others appear multiple times.

3. Final Answer:

   Rows with unique values only, no duplicates -> Option A

4. Quick Check:

   keep=False = Keep only unique rows [OK]

Hint: keep=False drops all duplicates, keeps only unique rows [OK]

Common Mistakes:

• Thinking keep=False keeps first or last duplicates
• Expecting some duplicates to remain
• Confusing output with keep='first' or keep='last'

4.

Consider this code snippet:

import pandas as pd

data = {'col': [1, 1, 2, 2, 3]}
df = pd.DataFrame(data)
result = df.drop_duplicates(keep='middle')
print(result)

What is the error in this code?

medium

A. DataFrame missing required columns

B. Invalid value for the 'keep' parameter

C. Syntax error in DataFrame creation

D. No error, code runs fine

Solution

1. Step 1: Check the 'keep' parameter values

   The valid values for 'keep' are 'first', 'last', or False. 'middle' is invalid.

2. Step 2: Identify the error raised by invalid 'keep'

   Using 'middle' causes a ValueError because pandas does not recognize it.

3. Final Answer:

   Invalid value for the 'keep' parameter -> Option B

4. Quick Check:

   keep must be 'first', 'last', or False [OK]

Hint: Use only 'first', 'last', or False for keep parameter [OK]

Common Mistakes:

• Using unsupported strings for keep
• Assuming 'middle' is valid
• Ignoring error messages

5.

You have a DataFrame df with duplicate rows based on column 'ID'. You want to remove all duplicates completely, so only rows with unique 'ID' remain. Which code snippet achieves this?

1) df.drop_duplicates(subset=['ID'], keep='first')
2) df.drop_duplicates(subset=['ID'], keep='last')
3) df.drop_duplicates(subset=['ID'], keep=False)
4) df.drop_duplicates()

hard

A. Option 1

B. Option 4

C. Option 2

D. Option 3

Solution

1. Step 1: Understand the goal

   The goal is to remove all duplicates so only unique 'ID' rows remain.

2. Step 2: Analyze each option

   Options 1 and 2 keep one duplicate (first or last). Option 4 removes duplicates but keeps first by default. Only option 3 with keep=False removes all duplicates completely.

3. Final Answer:

   Option 3 -> Option D

4. Quick Check:

   keep=False removes all duplicates [OK]

Hint: Use keep=False to remove all duplicates fully [OK]

Common Mistakes:

• Using keep='first' or 'last' when full removal needed
• Not specifying subset for duplicate check
• Assuming default drop_duplicates removes all duplicates