How to Remove Duplicates in Python: Simple Methods Explained

To remove duplicates in Python, you can convert a list to a set which automatically removes duplicates, then convert it back to a list. Alternatively, use a list comprehension with a helper set to keep order while removing duplicates.

Syntax

Here are common ways to remove duplicates from a list in Python:

• list(set(your_list)): Converts the list to a set to remove duplicates, then back to a list.
• [x for i, x in enumerate(your_list) if x not in your_list[:i]]: Keeps order by checking if the item appeared before.

python

unique_list = list(set(your_list))

unique_ordered = [x for i, x in enumerate(your_list) if x not in your_list[:i]]

Example

This example shows how to remove duplicates from a list while keeping the original order.

python

your_list = [1, 2, 2, 3, 4, 4, 5]

# Remove duplicates without order
unique_list = list(set(your_list))
print("Without order:", unique_list)

# Remove duplicates keeping order
unique_ordered = []
seen = set()
for item in your_list:
    if item not in seen:
        unique_ordered.append(item)
        seen.add(item)
print("With order:", unique_ordered)

Output

Without order: [1, 2, 3, 4, 5]
With order: [1, 2, 3, 4, 5]

Common Pitfalls

Using set() removes duplicates but does not keep the original order of items, which can be important in many cases.

Also, sets only work with hashable items, so if your list contains unhashable types like other lists, set() will cause an error.

python

your_list = [1, 2, 2, 3, 4, 4, 5]

# Wrong if order matters
unique_wrong = list(set(your_list))
print("Wrong order:", unique_wrong)

# Right way to keep order
unique_right = []
seen = set()
for item in your_list:
    if item not in seen:
        unique_right.append(item)
        seen.add(item)
print("Correct order:", unique_right)

Output

Wrong order: [1, 2, 3, 4, 5]
Correct order: [1, 2, 3, 4, 5]

Quick Reference

Summary tips for removing duplicates in Python:

• Use set() for quick duplicate removal when order does not matter.
• Use a loop with a seen set or list comprehension to keep order.
• Remember sets require hashable items.
• For Python 3.7+, dict.fromkeys() also preserves order while removing duplicates.

python

unique_ordered = list(dict.fromkeys(your_list))

Key Takeaways

Use set() to quickly remove duplicates but order is not preserved.

To keep order, use a loop with a seen set or dict.fromkeys().

Sets only work with hashable items; unhashable items cause errors.

List comprehensions with checks can remove duplicates while preserving order.

Python 3.7+ dictionaries keep insertion order, useful for duplicate removal.