How to Handle Duplicates in Python: Simple Fixes and Tips

To handle duplicates in Python, you can convert a list to a set to remove duplicates because sets only keep unique items. For ordered results, use a loop or dictionary to keep the first occurrence and skip duplicates.

Why This Happens

Duplicates happen when you have repeated items in a list or collection. Python lists allow duplicates by default, so if you add the same item multiple times, it stays there. This can cause problems if you want only unique items.

python

items = [1, 2, 2, 3, 4, 4, 4, 5]
print(items)

Output

[1, 2, 2, 3, 4, 4, 4, 5]

The Fix

To remove duplicates, convert the list to a set which keeps only unique values. If you want to keep the original order, use a loop with a helper set to track seen items and add only new ones.

python

items = [1, 2, 2, 3, 4, 4, 4, 5]

# Using set to remove duplicates (order not guaranteed)
unique_items = list(set(items))
print(unique_items)

# Keeping order while removing duplicates
seen = set()
unique_ordered = []
for item in items:
    if item not in seen:
        unique_ordered.append(item)
        seen.add(item)
print(unique_ordered)

Output

[1, 2, 3, 4, 5]
[1, 2, 3, 4, 5]

Prevention

To avoid duplicates, consider using data structures that do not allow duplicates like set or dict keys from the start. When adding items, check if they already exist. Use linting tools or code reviews to spot duplicate logic or data issues early.

Related Errors

Sometimes duplicates cause bugs like counting errors or wrong results. Similar issues include accidentally overwriting dictionary keys or adding duplicate rows in databases. Fixes usually involve checking for existence before adding or using unique constraints.

Key Takeaways

Use set to quickly remove duplicates from lists but note it does not keep order.

To keep order, track seen items with a set and build a new list.

Prevent duplicates by choosing the right data structure and checking before adding items.

Duplicates can cause logic errors, so handle them early in your code.

Use code reviews and linting to catch duplicate-related issues before runtime.