Removing duplicates in Power BI - Real Business Scenario

Scenario Mode

👤 Your Role: You are a sales analyst at a retail company.

📋 Request: Your manager wants a clean sales report without duplicate sales records to ensure accurate revenue analysis.

📊 Data: You have a sales dataset with columns: OrderID, Product, Customer, SalesAmount, and OrderDate. Some orders are duplicated due to system errors.

🎯 Deliverable: Produce a cleaned sales table in Power BI that removes duplicate orders based on OrderID, and create a report showing total sales by Product.

Sample Data

OrderID	Product	Customer	SalesAmount	OrderDate
1001	Notebook	Alice	25	2024-05-01
1002	Pen	Bob	5	2024-05-02
1003	Notebook	Charlie	25	2024-05-03
1002	Pen	Bob	5	2024-05-02
1004	Marker	Diana	10	2024-05-04
1005	Pen	Eva	5	2024-05-05
1003	Notebook	Charlie	25	2024-05-03
1006	Notebook	Frank	25	2024-05-06

1

Step 1: Load the sales data into Power BI Desktop from your source file.

Use 'Get Data' and select your data source file containing the sales data.

Expected Result

Sales data table loaded with all rows including duplicates.

2

Step 2: Open Power Query Editor to clean the data.

Click 'Transform Data' to open Power Query Editor.

Expected Result

Power Query Editor opens showing the sales data.

3

Step 3: Remove duplicate rows based on the OrderID column.

Select the 'OrderID' column, then click 'Remove Rows' > 'Remove Duplicates'.

Expected Result

Only unique OrderID rows remain; duplicate orders are removed.

4

Step 4: Close and apply the changes to load the cleaned data into Power BI.

Click 'Close & Apply' in Power Query Editor.

Expected Result

Cleaned sales data loaded into Power BI without duplicates.

5

Step 5: Create a report visual to show total sales by Product.

Insert a 'Clustered Column Chart'. Set Axis to 'Product' and Values to 'Sum of SalesAmount'.

Expected Result

Bar chart displays total sales amount for each product without double counting duplicates.

Final Result

Sales Report

Product    | Total Sales
-------------------------
Notebook   | ██████████ 75
Pen        | ████ 10
Marker     | ██ 10

(Bar chart with Notebook highest, Pen and Marker lower)

✓Duplicate orders were removed to avoid inflating sales numbers.

✓Notebook is the top-selling product with total sales of 75.

✓Pen and Marker have lower total sales after cleaning duplicates.

Bonus Challenge

Create a measure that counts the number of unique customers who bought each product after removing duplicates.

Show Hint

Use DAX function DISTINCTCOUNT on the Customer column filtered by Product.

Practice

1. What is the main purpose of removing duplicates in Power BI data?

easy

A. To add more rows to the dataset

B. To keep data clean and reliable by eliminating repeated rows

C. To change the data types of columns

D. To create new calculated columns

Solution

1. Step 1: Understand the concept of duplicates

   Duplicates are repeated rows that can cause errors or confusion in analysis.

2. Step 2: Purpose of removing duplicates

   Removing duplicates ensures the data is clean and reliable for accurate reporting.

3. Final Answer:

   To keep data clean and reliable by eliminating repeated rows -> Option B

4. Quick Check:

   Removing duplicates = Clean data [OK]

Hint: Duplicates cause errors; removing them cleans data fast [OK]

Common Mistakes:

• Thinking duplicates add useful data
• Confusing duplicates with data transformation
• Believing duplicates change data types

2. Which of the following is the correct syntax to remove duplicate rows from a table named Sales in Power Query?

easy

A. Table.RemoveDuplicates(Sales)

B. Remove.Duplicates(Sales)

C. Table.Distinct(Sales)

D. DistinctRows(Sales)

Solution

1. Step 1: Recall Power Query function for duplicates

   The correct function to remove duplicates is Table.Distinct.

2. Step 2: Apply function to the table

   Using Table.Distinct(Sales) removes duplicate rows from the Sales table.

3. Final Answer:

   Table.Distinct(Sales) -> Option C

4. Quick Check:

   Table.Distinct = Remove duplicates [OK]

Hint: Use Table.Distinct to remove duplicates in Power Query [OK]

Common Mistakes:

• Using non-existent functions like Remove.Duplicates
• Confusing syntax with other languages
• Adding extra parameters unnecessarily

3. Given the table Customers with columns CustomerID and City, what will be the result of Table.Distinct(Customers, {"CustomerID"})?

medium

A. Removes duplicate rows based on all columns

B. Removes duplicate rows based only on City column

C. Returns the original table without changes

D. Removes duplicate rows based only on CustomerID column

Solution

1. Step 1: Understand Table.Distinct with column list

   When columns are specified, duplicates are removed based on those columns only.

2. Step 2: Apply to CustomerID column

   Here, duplicates are removed based on CustomerID, ignoring other columns.

3. Final Answer:

   Removes duplicate rows based only on CustomerID column -> Option D

4. Quick Check:

   Distinct by columns = Remove duplicates by those columns [OK]

Hint: Specify columns in Table.Distinct to target duplicates [OK]

Common Mistakes:

• Assuming all columns are checked when columns are specified
• Confusing column names in the list
• Thinking it returns unchanged table

4. You wrote Table.Distinct(Sales, "ProductID") to remove duplicates by ProductID but get an error. What is the likely problem?

medium

A. The second argument should be a list of column names, not a single string

B. Table.Distinct does not accept a second argument

C. ProductID column does not exist in Sales table

D. You must use Table.RemoveDuplicates instead

Solution

1. Step 1: Check Table.Distinct syntax for columns

   The second argument must be a list of column names, e.g., {"ProductID"}, not a string.

2. Step 2: Identify error cause

   Passing "ProductID" as a string causes a syntax error because it expects a list.

3. Final Answer:

   The second argument should be a list of column names, not a single string -> Option A

4. Quick Check:

   Columns argument = List, not string [OK]

Hint: Use curly braces {} for column list in Table.Distinct [OK]

Common Mistakes:

• Passing column name as string instead of list
• Using wrong function name
• Assuming single argument works for columns

5. You have a table Orders with columns OrderID, CustomerID, and OrderDate. You want to keep only the first order per customer by date, removing duplicates of CustomerID but keeping the earliest OrderDate. Which approach is best in Power Query?

hard

A. Sort Orders by OrderDate ascending, then use Table.Distinct(Orders, {"CustomerID"})

B. Use Table.Distinct(Orders, {"CustomerID", "OrderDate"}) without sorting

C. Remove duplicates by OrderID column

D. Filter Orders to keep only rows where OrderDate is max per CustomerID

Solution

1. Step 1: Sort Orders by OrderDate ascending

   Sorting ensures earliest dates come first for each customer.

2. Step 2: Remove duplicates by CustomerID

   Using Table.Distinct(Orders, {"CustomerID"}) keeps the first row per customer, which is earliest due to sorting.

3. Final Answer:

   Sort Orders by OrderDate ascending, then use Table.Distinct(Orders, {"CustomerID"}) -> Option A

4. Quick Check:

   Sort then distinct = keep earliest per group [OK]

Hint: Sort by date first, then remove duplicates by customer [OK]

Common Mistakes:

• Removing duplicates without sorting first
• Including OrderDate in distinct columns
• Filtering by max date instead of min