drop_duplicates() for removal in Pandas - Mini Project: Build & Apply

Remove Duplicate Rows Using drop_duplicates()

📖 Scenario: You work in a small store that keeps track of sales data in a table. Sometimes, the same sale is accidentally recorded twice. You want to clean the data by removing these duplicate sales.

🎯 Goal: Build a small program that creates a sales data table, sets a column to check for duplicates, removes duplicate rows using drop_duplicates(), and prints the cleaned data.

📋 What You'll Learn

Create a pandas DataFrame called sales_data with exact columns and rows

Create a variable called subset_column to specify which column to check for duplicates

Use drop_duplicates() on sales_data with the subset parameter set to subset_column

Print the cleaned DataFrame

💡 Why This Matters

🌍 Real World

Cleaning duplicate records is a common task in data analysis to ensure accurate results.

💼 Career

Data scientists and analysts often need to clean data by removing duplicates before analysis or reporting.

1

Create the sales data DataFrame

Create a pandas DataFrame called sales_data with these exact columns and rows:
SaleID: [101, 102, 103, 102, 104]
Product: ['Apple', 'Banana', 'Apple', 'Banana', 'Orange']
Quantity: [5, 3, 5, 3, 2]

Pandas

import pandas as pd
# Create the sales_data DataFrame with the exact data
# Your code here

Hint

Use pd.DataFrame and pass a dictionary with keys as column names and values as lists of data.

2

Set the column to check for duplicates

Create a variable called subset_column and set it to the string 'SaleID' to specify which column to check for duplicates.

Pandas

import pandas as pd
sales_data = pd.DataFrame({
    'SaleID': [101, 102, 103, 102, 104],
    'Product': ['Apple', 'Banana', 'Apple', 'Banana', 'Orange'],
    'Quantity': [5, 3, 5, 3, 2]
})

# Create subset_column variable with value 'SaleID'
# Your code here

Hint

Just assign the string 'SaleID' to the variable subset_column.

3

Remove duplicate rows using drop_duplicates()

Create a new DataFrame called cleaned_data by using sales_data.drop_duplicates() with the subset parameter set to subset_column.

Pandas

import pandas as pd
sales_data = pd.DataFrame({
    'SaleID': [101, 102, 103, 102, 104],
    'Product': ['Apple', 'Banana', 'Apple', 'Banana', 'Orange'],
    'Quantity': [5, 3, 5, 3, 2]
})
subset_column = 'SaleID'

# Use drop_duplicates() on sales_data with subset=subset_column and save to cleaned_data
# Your code here

Hint

Use drop_duplicates() on sales_data and pass subset=subset_column.

4

Print the cleaned data

Print the cleaned_data DataFrame to see the sales data after removing duplicates.

Pandas

import pandas as pd
sales_data = pd.DataFrame({
    'SaleID': [101, 102, 103, 102, 104],
    'Product': ['Apple', 'Banana', 'Apple', 'Banana', 'Orange'],
    'Quantity': [5, 3, 5, 3, 2]
})
subset_column = 'SaleID'
cleaned_data = sales_data.drop_duplicates(subset=subset_column)

# Print the cleaned_data DataFrame
# Your code here

Hint

Use print(cleaned_data) to show the cleaned table.

Practice

1. What does the drop_duplicates() function do in pandas?

easy

A. Removes repeated rows from a DataFrame

B. Adds new rows to a DataFrame

C. Sorts the DataFrame by column values

D. Changes the data type of columns

Solution

1. Step 1: Understand the purpose of drop_duplicates()

   This function is designed to remove rows that appear more than once in a DataFrame.

2. Step 2: Compare options with function behavior

   Only Removes repeated rows from a DataFrame correctly describes removing repeated rows. Other options describe different functions.

3. Final Answer:

   Removes repeated rows from a DataFrame -> Option A

4. Quick Check:

   drop_duplicates() removes repeats [OK]

Hint: Remember: drop_duplicates() cleans repeated rows fast [OK]

Common Mistakes:

• Confusing drop_duplicates() with sorting functions
• Thinking it adds or modifies data instead of removing duplicates
• Assuming it changes data types

2. Which of the following is the correct syntax to remove duplicate rows from a DataFrame df?

easy

A. df.drop_duplicate_rows()

B. df.remove_duplicates()

C. df.delete_duplicates()

D. df.drop_duplicates()

Solution

1. Step 1: Recall the exact pandas method name

   The correct method to remove duplicates is drop_duplicates().

2. Step 2: Check each option for correct syntax

   Only df.drop_duplicates() uses the correct method name and syntax. Others are invalid pandas methods.

3. Final Answer:

   df.drop_duplicates() -> Option D

4. Quick Check:

   Correct method name is drop_duplicates() [OK]

Hint: Use exact method name: drop_duplicates() [OK]

Common Mistakes:

• Using incorrect method names like remove_duplicates()
• Adding extra words like delete or rows
• Confusing with other DataFrame methods

3. Given the DataFrame df below:

import pandas as pd
data = {'A': [1, 2, 2, 3], 'B': ['x', 'y', 'y', 'z']}
df = pd.DataFrame(data)
print(df.drop_duplicates())

What will be the output?

medium

A. Rows with index 1, 2, and 3

B. Rows with index 0, 1, and 3

C. Rows with index 0, 2, and 3

D. All rows remain unchanged

Solution

1. Step 1: Identify duplicate rows in the DataFrame

   Rows 2 and 3 have the same values (A=2, B='y'). So one will be removed.

2. Step 2: Understand default behavior of drop_duplicates()

   By default, it keeps the first occurrence and removes later duplicates. So row 1 stays, row 2 is removed.

3. Final Answer:

   Rows with index 0, 1, and 3 -> Option B

4. Quick Check:

   First duplicates kept, later removed [OK]

Hint: drop_duplicates() keeps first by default [OK]

Common Mistakes:

• Assuming last duplicate is kept instead of first
• Thinking all duplicates are removed including first
• Confusing row indices after removal

4. What is wrong with this code snippet to remove duplicates from DataFrame df?

df.drop_duplicates(inplace=True)
print(df.drop_duplicates())

medium

A. Calling drop_duplicates() twice is redundant

B. Using inplace=True and then printing drop_duplicates() returns original data

C. drop_duplicates() does not have inplace parameter

D. print statement syntax is incorrect

Solution

1. Step 1: Understand inplace=True effect

   Using inplace=True modifies df directly by removing duplicates.

2. Step 2: Analyze print statement

   After the first call, df is already deduplicated. The second call to df.drop_duplicates() is redundant as it returns the same DataFrame since there are no duplicates left.

3. Final Answer:

   Calling drop_duplicates() twice is redundant -> Option A

4. Quick Check:

   Second call redundant after inplace [OK]

Hint: inplace=True changes df; extra drop_duplicates() returns copy [OK]

Common Mistakes:

• Thinking drop_duplicates() lacks inplace parameter
• Assuming print statement is wrong syntax
• Believing calling twice causes error

5. You have a DataFrame df with columns ['Name', 'Age', 'City']. You want to remove duplicates only based on Name and City, keeping the last occurrence. Which code achieves this?

hard

A. df.drop_duplicates(columns=['Name', 'City'], keep='last')

B. df.drop_duplicates(subset=['Name', 'City'], keep='first')

C. df.drop_duplicates(subset=['Name', 'City'], keep='last')

D. df.drop_duplicates(subset=['Age'], keep='last')

Solution

1. Step 1: Use subset parameter to specify columns

   To check duplicates only on 'Name' and 'City', use subset=['Name', 'City'].

2. Step 2: Set keep='last' to keep last occurrence

   keep='last' keeps the last duplicate row, removing earlier ones.

3. Final Answer:

   df.drop_duplicates(subset=['Name', 'City'], keep='last') -> Option C

4. Quick Check:

   subset selects columns; keep='last' keeps last duplicate [OK]

Hint: Use subset for columns, keep='last' to keep last duplicate [OK]

Common Mistakes:

• Using 'columns' instead of 'subset' parameter
• Forgetting to set keep='last' to keep last occurrence
• Selecting wrong columns for subset