Keeping first vs last vs none in Pandas - Hands-On Comparison

Keeping First vs Last vs None in pandas

📖 Scenario: You work in a store's data team. You have a list of sales records with some duplicate entries for the same product. You want to clean the data by removing duplicates but keep either the first sale, the last sale, or remove all duplicates completely.

🎯 Goal: Learn how to use pandas drop_duplicates() with keep='first', keep='last', and keep=False options to control which duplicates to keep or remove.

📋 What You'll Learn

Create a pandas DataFrame called sales with given data

Create a variable called subset_cols to specify columns to check duplicates

Use drop_duplicates() with keep='first' to keep first duplicates

Use drop_duplicates() with keep='last' to keep last duplicates

Use drop_duplicates() with keep=False to remove all duplicates

Print the resulting DataFrames

💡 Why This Matters

🌍 Real World

Cleaning duplicate sales records is common in retail data analysis to ensure accurate reporting and inventory management.

💼 Career

Data analysts and data scientists often need to remove or handle duplicates in datasets before analysis or modeling.

1

Create the sales DataFrame

Create a pandas DataFrame called sales with these exact columns and rows:
{'Product': ['Apple', 'Banana', 'Apple', 'Banana', 'Cherry', 'Apple'], 'Price': [100, 80, 100, 90, 120, 100], 'Quantity': [5, 7, 5, 8, 10, 5]}

Pandas

import pandas as pd
# Create the sales DataFrame with the given data
# Your code here

Hint

Use pd.DataFrame() with a dictionary of lists for columns.

2

Create subset_cols variable for duplicate check

Create a variable called subset_cols and set it to a list containing the columns 'Product' and 'Price' to check duplicates based on these columns.

Pandas

import pandas as pd
sales = pd.DataFrame({'Product': ['Apple', 'Banana', 'Apple', 'Banana', 'Cherry', 'Apple'], 'Price': [100, 80, 100, 90, 120, 100], 'Quantity': [5, 7, 5, 8, 10, 5]})
# Create subset_cols list with 'Product' and 'Price'
# Your code here

Hint

Just assign the list ['Product', 'Price'] to subset_cols.

3

Remove duplicates keeping the first occurrence

Create a new DataFrame called keep_first by using sales.drop_duplicates() with subset=subset_cols and keep='first' to keep the first duplicate rows.

Pandas

import pandas as pd
sales = pd.DataFrame({'Product': ['Apple', 'Banana', 'Apple', 'Banana', 'Cherry', 'Apple'], 'Price': [100, 80, 100, 90, 120, 100], 'Quantity': [5, 7, 5, 8, 10, 5]})
subset_cols = ['Product', 'Price']
# Create keep_first DataFrame by dropping duplicates keeping first
# Your code here

Hint

Use drop_duplicates() with subset=subset_cols and keep='first'.

4

Remove duplicates keeping the last occurrence and removing all duplicates

Create two new DataFrames:
1. keep_last by dropping duplicates with subset=subset_cols and keep='last'.
2. keep_none by dropping duplicates with subset=subset_cols and keep=False to remove all duplicates.
Then print keep_first, keep_last, and keep_none.

Pandas

import pandas as pd
sales = pd.DataFrame({'Product': ['Apple', 'Banana', 'Apple', 'Banana', 'Cherry', 'Apple'], 'Price': [100, 80, 100, 90, 120, 100], 'Quantity': [5, 7, 5, 8, 10, 5]})
subset_cols = ['Product', 'Price']
keep_first = sales.drop_duplicates(subset=subset_cols, keep='first')
# Create keep_last DataFrame by dropping duplicates keeping last
# Create keep_none DataFrame by dropping duplicates keeping none
# Print keep_first, keep_last, and keep_none
# Your code here

Hint

Use drop_duplicates() with keep='last' and keep=False. Then print all three DataFrames.

Practice

1.

In pandas, what does keep='first' do when removing duplicates?

easy

A. Keeps the first occurrence of each duplicate and removes the rest

B. Keeps the last occurrence of each duplicate and removes the rest

C. Removes all duplicates, keeping only unique rows

D. Keeps all duplicates without removing any

Solution

1. Step 1: Understand the meaning of keep='first'

   This option keeps the first row of each set of duplicates and removes the later ones.

2. Step 2: Compare with other options

   Unlike keep='last' which keeps the last occurrence, keep='first' keeps the earliest one.

3. Final Answer:

   Keeps the first occurrence of each duplicate and removes the rest -> Option A

4. Quick Check:

   keep='first' = Keep first duplicate [OK]

Hint: Remember: 'first' keeps earliest duplicate row [OK]

Common Mistakes:

• Confusing keep='first' with keep='last'
• Thinking keep='first' removes all duplicates
• Assuming keep='first' keeps all duplicates

2.

Which of the following is the correct syntax to remove duplicates in a pandas DataFrame df keeping only the last occurrence?

df._____('column_name', keep='last')

easy

A. drop_dupes

B. remove_duplicates

C. drop_duplicates

D. delete_duplicates

Solution

1. Step 1: Identify the pandas method for removing duplicates

   The correct method is drop_duplicates to remove duplicate rows.

2. Step 2: Confirm syntax for keeping last occurrence

   Using keep='last' inside drop_duplicates keeps the last duplicate row.

3. Final Answer:

   drop_duplicates -> Option C

4. Quick Check:

   Method to remove duplicates = drop_duplicates [OK]

Hint: Use df.drop_duplicates() to remove duplicates [OK]

Common Mistakes:

• Using non-existent methods like remove_duplicates
• Confusing method names with similar words
• Forgetting to specify keep='last'

3.

What will be the output of the following code?

import pandas as pd

data = {'A': [1, 2, 2, 3, 3, 3], 'B': ['x', 'y', 'y', 'z', 'z', 'z']}
df = pd.DataFrame(data)
result = df.drop_duplicates(keep=False)
print(result)

medium

A. Rows with unique values only, no duplicates

B. All rows except the first occurrence of duplicates

C. Rows with A=1 B='x', A=2 B='y', A=3 B='z' only

D. Empty DataFrame

Solution

1. Step 1: Understand keep=False behavior

   Using keep=False removes all duplicates, keeping only rows that appear once.

2. Step 2: Identify unique rows in the DataFrame

   In the data, only the row with A=1 and B='x' is unique; others appear multiple times.

3. Final Answer:

   Rows with unique values only, no duplicates -> Option A

4. Quick Check:

   keep=False = Keep only unique rows [OK]

Hint: keep=False drops all duplicates, keeps only unique rows [OK]

Common Mistakes:

• Thinking keep=False keeps first or last duplicates
• Expecting some duplicates to remain
• Confusing output with keep='first' or keep='last'

4.

Consider this code snippet:

import pandas as pd

data = {'col': [1, 1, 2, 2, 3]}
df = pd.DataFrame(data)
result = df.drop_duplicates(keep='middle')
print(result)

What is the error in this code?

medium

A. DataFrame missing required columns

B. Invalid value for the 'keep' parameter

C. Syntax error in DataFrame creation

D. No error, code runs fine

Solution

1. Step 1: Check the 'keep' parameter values

   The valid values for 'keep' are 'first', 'last', or False. 'middle' is invalid.

2. Step 2: Identify the error raised by invalid 'keep'

   Using 'middle' causes a ValueError because pandas does not recognize it.

3. Final Answer:

   Invalid value for the 'keep' parameter -> Option B

4. Quick Check:

   keep must be 'first', 'last', or False [OK]

Hint: Use only 'first', 'last', or False for keep parameter [OK]

Common Mistakes:

• Using unsupported strings for keep
• Assuming 'middle' is valid
• Ignoring error messages

5.

You have a DataFrame df with duplicate rows based on column 'ID'. You want to remove all duplicates completely, so only rows with unique 'ID' remain. Which code snippet achieves this?

1) df.drop_duplicates(subset=['ID'], keep='first')
2) df.drop_duplicates(subset=['ID'], keep='last')
3) df.drop_duplicates(subset=['ID'], keep=False)
4) df.drop_duplicates()

hard

A. Option 1

B. Option 4

C. Option 2

D. Option 3

Solution

1. Step 1: Understand the goal

   The goal is to remove all duplicates so only unique 'ID' rows remain.

2. Step 2: Analyze each option

   Options 1 and 2 keep one duplicate (first or last). Option 4 removes duplicates but keeps first by default. Only option 3 with keep=False removes all duplicates completely.

3. Final Answer:

   Option 3 -> Option D

4. Quick Check:

   keep=False removes all duplicates [OK]

Hint: Use keep=False to remove all duplicates fully [OK]

Common Mistakes:

• Using keep='first' or 'last' when full removal needed
• Not specifying subset for duplicate check
• Assuming default drop_duplicates removes all duplicates