Finding duplicates efficiently in SQL - Step-by-Step Execution

Concept Flow - Finding duplicates efficiently

Start with table data

↓

Group rows by target column(s)

↓

Count rows in each group

↓

Filter groups where count > 1

↓

Return duplicate values and counts

↓

End

This flow groups data by the column(s) to check duplicates, counts each group, then filters to keep only those with more than one entry.

Execution Sample

SQL

SELECT column_name, COUNT(*) AS count
FROM table_name
GROUP BY column_name
HAVING COUNT(*) > 1;

This query finds duplicate values in 'column_name' by grouping and counting, then selecting groups with more than one row.

Execution Table

Step	Action	Group By Result	Count	Filter Condition	Output
1	Scan all rows in table	N/A	N/A	N/A	All rows read
2	Group rows by column_name	Groups: A, B, C, A, B, A	N/A	N/A	Groups formed
3	Count rows in each group	A, B, C	3, 2, 1	N/A	Counts calculated
4	Filter groups with count > 1	A, B	3, 2	True	Groups A and B kept
5	Return duplicates with counts	A, B	3, 2	N/A	Output rows: (A,3), (B,2)
6	End	N/A	N/A	N/A	Query complete

💡 All groups processed; only those with count > 1 returned as duplicates.

Variable Tracker

Variable	Start	After Step 2	After Step 3	After Step 4	Final
Groups	None	A, B, C, A, B, A	A, B, C	A, B	A, B
Counts	None	N/A	3, 2, 1	3, 2	3, 2
Output	None	None	None	None	(A,3), (B,2)

Key Moments - 3 Insights

Why do we use GROUP BY before HAVING?

GROUP BY collects rows into groups by the column value, so HAVING can filter groups based on aggregate functions like COUNT. Without GROUP BY, HAVING cannot filter duplicates.

Why do we use HAVING COUNT(*) > 1 instead of WHERE?

WHERE filters rows before grouping, but duplicates are identified after grouping. HAVING filters groups based on aggregate results like COUNT, so it's needed here.

What if the column has NULL values? Are they counted as duplicates?

Yes, NULLs are grouped together by GROUP BY and counted. If multiple NULLs exist, they appear as duplicates in the output.

Visual Quiz - 3 Questions

Look at the execution table, what is the count for group 'B' after Step 3?

A 1

B 3

C 2

D 0

💡 Hint

Check the 'Count' column in Step 3 row in the execution_table.

At which step are groups with only one row removed?

A Step 4

B Step 3

C Step 2

D Step 5

💡 Hint

Look at the 'Filter groups with count > 1' action in the execution_table.

If the table had no duplicates, what would the output be after Step 5?

A All groups with count 1

B Empty result set

C All rows from the table

D Error message

💡 Hint

Recall that HAVING COUNT(*) > 1 filters out groups with count 1 or less.

Concept Snapshot

Finding duplicates efficiently:
Use GROUP BY on the column(s) to group rows.
Use COUNT(*) to count rows per group.
Use HAVING COUNT(*) > 1 to keep only duplicates.
Returns duplicate values and their counts.
Works even if NULLs are present.

Full Transcript

To find duplicates efficiently in SQL, we group rows by the column we want to check duplicates in. Then we count how many rows are in each group. Groups with more than one row mean duplicates exist. We use HAVING COUNT(*) > 1 to filter these groups. The query returns the duplicate values and how many times they appear. This method works well even if the column has NULL values, as NULLs are grouped together. The process starts by scanning all rows, grouping them, counting each group, filtering groups with count greater than one, and finally returning those duplicates.

Practice

1. What is the main purpose of using GROUP BY with HAVING COUNT(*) > 1 in SQL?

easy

A. To sort the table alphabetically

B. To find duplicate values in a column

C. To delete rows from a table

D. To create a new table

Solution

1. Step 1: Understand GROUP BY usage

   GROUP BY groups rows that have the same value in specified columns.

2. Step 2: Use HAVING COUNT(*) > 1 to filter groups

   HAVING COUNT(*) > 1 keeps only groups with more than one row, meaning duplicates.

3. Final Answer:

   To find duplicate values in a column -> Option B

4. Quick Check:

   GROUP BY + HAVING COUNT(*) > 1 = Find duplicates [OK]

Hint: Use GROUP BY with HAVING COUNT(*) > 1 to spot duplicates fast [OK]

Common Mistakes:

• Confusing HAVING with WHERE clause
• Using COUNT(column) instead of COUNT(*)
• Not grouping by the correct column

2. Which of the following SQL queries correctly finds duplicate email values in a table named users?

easy

A. SELECT email FROM users GROUP BY email HAVING COUNT(*) > 1;

B. SELECT email FROM users WHERE COUNT(email) > 1;

C. SELECT email FROM users GROUP BY email WHERE COUNT(*) > 1;

D. SELECT email FROM users HAVING COUNT(email) > 1;

Solution

1. Step 1: Check correct syntax for grouping and filtering

   GROUP BY groups rows; HAVING filters groups. WHERE cannot filter aggregates.

2. Step 2: Identify correct query

   SELECT email FROM users GROUP BY email HAVING COUNT(*) > 1; uses GROUP BY email and HAVING COUNT(*) > 1 correctly.

3. Final Answer:

   SELECT email FROM users GROUP BY email HAVING COUNT(*) > 1; -> Option A

4. Quick Check:

   GROUP BY + HAVING filters duplicates correctly [OK]

Hint: Use HAVING, not WHERE, to filter grouped results [OK]

Common Mistakes:

• Using WHERE with aggregate functions
• Placing WHERE after GROUP BY
• Using COUNT(column) incorrectly in WHERE

3. Given the table orders with column customer_id, what will this query return?

SELECT customer_id FROM orders GROUP BY customer_id HAVING COUNT(*) > 2;

medium

A. All customer IDs with exactly 2 orders

B. All customer IDs regardless of order count

C. All customer IDs with less than 2 orders

D. All customer IDs with more than 2 orders

Solution

1. Step 1: Understand HAVING COUNT(*) > 2 condition

   This filters groups where the count of rows is greater than 2.

2. Step 2: Apply to customer_id groups

   Only customer IDs with more than 2 orders appear in the result.

3. Final Answer:

   All customer IDs with more than 2 orders -> Option D

4. Quick Check:

   HAVING COUNT(*) > 2 means count > 2 [OK]

Hint: HAVING COUNT(*) > N finds groups with more than N rows [OK]

Common Mistakes:

• Thinking it finds exactly 2 instead of more than 2
• Confusing WHERE and HAVING clauses
• Ignoring GROUP BY effect

4. Identify the error in this query to find duplicate username values:

SELECT username FROM users WHERE COUNT(username) > 1 GROUP BY username;

medium

A. Missing ORDER BY clause

B. GROUP BY should come before WHERE

C. COUNT() cannot be used in WHERE clause

D. COUNT() should be COUNT(DISTINCT username)

Solution

1. Step 1: Check WHERE clause usage

   WHERE cannot filter on aggregate functions like COUNT().

2. Step 2: Correct clause to use HAVING

   HAVING is used to filter groups after GROUP BY with aggregates.

3. Final Answer:

   COUNT() cannot be used in WHERE clause -> Option C

4. Quick Check:

   Aggregates need HAVING, not WHERE [OK]

Hint: Use HAVING, not WHERE, to filter aggregates [OK]

Common Mistakes:

• Using WHERE with aggregate functions
• Placing GROUP BY after WHERE
• Confusing COUNT() usage

5. You want to find duplicate product_code values in a large inventory table but also want to see how many times each duplicate appears. Which query is the most efficient and correct?

hard

A. SELECT product_code, COUNT(*) AS count FROM inventory GROUP BY product_code HAVING count > 1;

B. SELECT product_code, COUNT(*) AS count FROM inventory WHERE COUNT(*) > 1 GROUP BY product_code;

C. SELECT product_code FROM inventory GROUP BY product_code HAVING COUNT(*) > 1 ORDER BY product_code;

D. SELECT product_code, COUNT(product_code) FROM inventory GROUP BY product_code WHERE COUNT(product_code) > 1;

Solution

1. Step 1: Check correct use of GROUP BY and HAVING

   SELECT product_code, COUNT(*) AS count FROM inventory GROUP BY product_code HAVING count > 1; groups by product_code and filters groups with count > 1 using HAVING, which is correct.

2. Step 2: Verify alias usage and efficiency

   Using COUNT(*) AS count allows filtering with HAVING count > 1 efficiently and shows counts.

3. Final Answer:

   SELECT product_code, COUNT(*) AS count FROM inventory GROUP BY product_code HAVING count > 1; -> Option A

4. Quick Check:

   GROUP BY + HAVING with alias count shows duplicates and counts [OK]

Hint: Use HAVING with COUNT(*) alias to filter and show duplicates count [OK]

Common Mistakes:

• Using WHERE with COUNT()
• Placing WHERE after GROUP BY
• Not selecting count to see duplicate frequency