duplicated() for finding duplicates in Pandas - Mini Project: Build & Apply

Using duplicated() to Find Duplicate Rows in Data

📖 Scenario: You work in a store's data team. You have a list of sales records. Sometimes, the same sale is recorded twice by mistake. You want to find these duplicate sales to fix the data.

🎯 Goal: You will create a small sales data table, then use pandas duplicated() to find which rows are duplicates.

📋 What You'll Learn

Create a pandas DataFrame called sales with given sales data

Create a variable keep_option to decide which duplicates to mark

Use duplicated() on sales with keep=keep_option to find duplicates

Print the boolean Series showing duplicate rows

💡 Why This Matters

🌍 Real World

Duplicate data can cause errors in reports and decisions. Finding duplicates helps keep data clean and trustworthy.

💼 Career

Data analysts and scientists often clean data by identifying and handling duplicates to ensure accurate analysis.

1

Create the sales DataFrame

Import pandas as pd and create a DataFrame called sales with these exact rows: {'Product': ['Apple', 'Banana', 'Apple', 'Banana', 'Apple'], 'Price': [1.0, 0.5, 1.0, 0.5, 1.0], 'Quantity': [10, 5, 10, 5, 10]}

Pandas

import pandas as pd
# Create the sales DataFrame with the exact data
# Your code here

Hint

Use pd.DataFrame() with a dictionary of lists for columns.

2

Set the keep option for duplicates

Create a variable called keep_option and set it to the string 'first' to mark duplicates except the first occurrence.

Pandas

import pandas as pd
sales = pd.DataFrame({
    'Product': ['Apple', 'Banana', 'Apple', 'Banana', 'Apple'],
    'Price': [1.0, 0.5, 1.0, 0.5, 1.0],
    'Quantity': [10, 5, 10, 5, 10]
})
# Set keep_option to 'first'
# Your code here

Hint

The keep parameter in duplicated() can be 'first', 'last', or False.

3

Find duplicate rows using duplicated()

Create a variable called duplicates that stores the result of sales.duplicated(keep=keep_option) to find duplicate rows.

Pandas

import pandas as pd
sales = pd.DataFrame({
    'Product': ['Apple', 'Banana', 'Apple', 'Banana', 'Apple'],
    'Price': [1.0, 0.5, 1.0, 0.5, 1.0],
    'Quantity': [10, 5, 10, 5, 10]
})
keep_option = 'first'
# Use duplicated() with keep=keep_option to find duplicates
# Your code here

Hint

Call duplicated() on the DataFrame with the keep argument.

4

Print the duplicates boolean Series

Write a print statement to display the duplicates variable.

Pandas

import pandas as pd
sales = pd.DataFrame({
    'Product': ['Apple', 'Banana', 'Apple', 'Banana', 'Apple'],
    'Price': [1.0, 0.5, 1.0, 0.5, 1.0],
    'Quantity': [10, 5, 10, 5, 10]
})
keep_option = 'first'
duplicates = sales.duplicated(keep=keep_option)
# Print the duplicates Series
# Your code here

Hint

Use print(duplicates) to see which rows are duplicates.

Practice

1. What does the duplicated() function in pandas do?

easy

A. It identifies duplicate rows in a DataFrame.

B. It removes duplicate rows from a DataFrame.

C. It sorts the DataFrame by duplicate values.

D. It counts the number of unique rows in a DataFrame.

Solution

1. Step 1: Understand the purpose of duplicated()

   The duplicated() function checks each row and marks it as True if it is a duplicate of a previous row.

2. Step 2: Differentiate from other functions

   Unlike functions that remove duplicates or count unique rows, duplicated() only flags duplicates without changing the DataFrame.

3. Final Answer:

   It identifies duplicate rows in a DataFrame. -> Option A

4. Quick Check:

   duplicated() = identifies duplicates [OK]

Hint: Remember: duplicated() flags duplicates, doesn't remove them [OK]

Common Mistakes:

• Confusing duplicated() with drop_duplicates()
• Thinking duplicated() sorts data
• Assuming duplicated() counts unique rows

2. Which of the following is the correct syntax to find duplicates in a pandas DataFrame df?

easy

A. df.duplicated()

B. df.duplicates()

C. df.find_duplicates()

D. df.duplicate()

Solution

1. Step 1: Recall the exact function name

   The correct pandas function to find duplicate rows is duplicated(), not any other variation.

2. Step 2: Verify syntax correctness

   The function is called on the DataFrame object as df.duplicated() with parentheses.

3. Final Answer:

   df.duplicated() -> Option A

4. Quick Check:

   Correct function name = df.duplicated() [OK]

Hint: Use exact function name: duplicated() with parentheses [OK]

Common Mistakes:

• Misspelling the function name
• Omitting parentheses after duplicated
• Using non-existent functions like duplicates()

3. Given the DataFrame df below, what is the output of df.duplicated()?

import pandas as pd
data = {'A': [1, 2, 2, 3], 'B': ['x', 'y', 'y', 'z']}
df = pd.DataFrame(data)
print(df.duplicated())

medium

A. [True, False, True, False]

B. [False, False, True, False]

C. [False, True, False, True]

D. [False, False, False, True]

Solution

1. Step 1: Identify duplicate rows in the DataFrame

   The rows are: 0: (1, 'x'), 1: (2, 'y'), 2: (2, 'y'), 3: (3, 'z'). Row 2 is a duplicate of row 1.

2. Step 2: Understand duplicated() output

   duplicated() returns False for the first occurrence and True for duplicates. So, rows 0,1,3 are False; row 2 is True.

3. Final Answer:

   [False, False, True, False] -> Option B

4. Quick Check:

   First duplicate row marked True = [False, False, True, False] [OK]

Hint: First occurrence is False, duplicates after are True [OK]

Common Mistakes:

• Marking first occurrence as True
• Confusing row indices with values
• Assuming all duplicates are True regardless of position

4. What is wrong with this code snippet to find duplicates in column 'A' of DataFrame df?

duplicates = df.duplicated['A']
print(duplicates)

medium

A. duplicated() cannot check specific columns

B. Missing import statement for pandas

C. The print statement is incorrect

D. Using square brackets instead of parentheses to call duplicated()

Solution

1. Step 1: Check function call syntax

   The code uses square brackets [] instead of parentheses () to call the duplicated() method, which is incorrect syntax.

2. Step 2: Correct way to specify columns

   To check duplicates based on column 'A', the correct syntax is df.duplicated(subset=['A']) with parentheses.

3. Final Answer:

   Using square brackets instead of parentheses to call duplicated() -> Option D

4. Quick Check:

   Call methods with () not [] [OK]

Hint: Call duplicated() with parentheses, not square brackets [OK]

Common Mistakes:

• Using [] instead of () to call functions
• Trying to access duplicated as an attribute
• Not using subset parameter for specific columns

5. You have a DataFrame df with columns 'Name', 'Age', and 'City'. You want to find duplicates only based on 'Name' and 'City', ignoring 'Age'. Which code correctly finds these duplicates?

hard

A. df.duplicated(['Name', 'City'], keep=True)

B. df.duplicated(columns=['Name', 'City'])

C. df.duplicated(subset=['Name', 'City'])

D. df.duplicated('Name', 'City')

Solution

1. Step 1: Use subset parameter to specify columns

   To check duplicates based on specific columns, use the subset parameter with a list of column names.

2. Step 2: Validate correct syntax

   The correct syntax is df.duplicated(subset=['Name', 'City']). df.duplicated(columns=['Name', 'City']) uses a wrong parameter name (columns), and options A and C use invalid syntax.

3. Final Answer:

   df.duplicated(subset=['Name', 'City']) -> Option C

4. Quick Check:

   Use subset=['col1', 'col2'] to check specific columns [OK]

Hint: Use subset parameter to check duplicates on chosen columns [OK]

Common Mistakes:

• Using wrong parameter name like columns instead of subset
• Passing list without parameter name
• Confusing keep parameter with subset