PythonHow-ToBeginner · 3 min read

How to Remove Duplicates from List in Python Quickly

To remove duplicates from a list in Python, convert the list to a set which automatically removes duplicates, then convert it back to a list. Alternatively, use a loop or dictionary to preserve order while removing duplicates.

📐

Syntax

You can remove duplicates by converting a list to a set and back to a list. The syntax is:

unique_list = list(set(your_list)) - removes duplicates but does not keep order.
To keep order, use a loop or dictionary:
unique_list = list(dict.fromkeys(your_list))

python

your_list = [1, 2, 2, 3, 4, 4, 5]
unique_list = list(set(your_list))  # Removes duplicates but order may change

# To keep order:
unique_ordered = list(dict.fromkeys(your_list))

💻

Example

This example shows how to remove duplicates from a list using both methods: converting to a set and using dict.fromkeys() to keep the original order.

python

your_list = [3, 1, 2, 3, 4, 2, 5, 1]

# Remove duplicates without preserving order
unique_list = list(set(your_list))
print("Without order:", unique_list)

# Remove duplicates and preserve order
unique_ordered = list(dict.fromkeys(your_list))
print("With order:", unique_ordered)

Output

Without order: [1, 2, 3, 4, 5] With order: [3, 1, 2, 4, 5]

⚠️

Common Pitfalls

One common mistake is using set() to remove duplicates when the order of items matters, because sets do not keep order. Another is trying to remove duplicates by looping and modifying the list at the same time, which can cause errors or unexpected results.

Always use dict.fromkeys() or a loop that builds a new list to keep order safely.

python

your_list = [1, 2, 2, 3]

# Wrong: modifying list while looping (can cause bugs)
for item in your_list[:]:
    if your_list.count(item) > 1:
        your_list.remove(item)
print("Wrong way result:", your_list)

# Right: use dict.fromkeys() to keep order and remove duplicates
unique_ordered = list(dict.fromkeys([1, 2, 2, 3]))
print("Right way result:", unique_ordered)

Output

Wrong way result: [1, 2, 3] Right way result: [1, 2, 3]

📊

Quick Reference

Here is a quick summary of methods to remove duplicates from a list in Python:

Method	Description	Keeps Order?
`list(set(your_list))`	Converts list to set and back, removes duplicates	No
`list(dict.fromkeys(your_list))`	Uses dictionary keys to remove duplicates	Yes
Loop with new list	Manually add items if not already added	Yes

✅

Key Takeaways

Use list(set(your_list)) to quickly remove duplicates but order is not preserved.

Use list(dict.fromkeys(your_list)) to remove duplicates while keeping the original order.

Avoid modifying a list while looping over it to remove duplicates; it causes bugs.

Dictionaries in Python 3.7+ keep insertion order, making dict.fromkeys() a simple solution.

Choose the method based on whether you need to keep the order of items.