Remove Duplicate Items from a Python List

You will learn how to identify duplicates in a Python list and create a new list containing unique items. You will also understand when to use a set, when order matters, and how to handle values that cannot be placed in a set.

A duplicate is a value that appears more than once in a list.

numbers = [4, 7, 4, 2, 7]

In this list, 4 and 7 are duplicates because each occurs more than once.

Python's set type stores each distinct value only once. That makes it useful for checking whether duplicates exist:

has_duplicates = len(numbers) != len(set(numbers))

If duplicates exist, converting the list to a set makes it shorter. However, a plain set should not be used when the original item order must be preserved. In many programs, order is meaningful—for example, showing tags in the order a user entered them or processing records in their original sequence.

A common order-preserving solution keeps a seen set and builds a new list. The set provides fast membership checks, while the list keeps the output order.

Imagine a guest list at an event.

• The original list is everyone who arrives, including people who accidentally appear twice.
• The seen set is a check-in register. It records who has already entered.
• The unique output list is the final guest list, written in arrival order.

For each item, ask: “Have I seen this before?”

• If yes, skip it.
• If no, add it to both the register and the final list.

Use set() for a quick duplicate check:

items = ["apple", "banana", "apple"]

has_duplicates = len(items) != len(set(items))
print(has_duplicates)  # True

To remove duplicates while preserving the first occurrence of every item, use dict.fromkeys() in modern Python:

items = ["apple", "banana", "apple", "orange", "banana"]
unique_items = list(dict.fromkeys(items))

print(unique_items)
# ['apple', 'banana', 'orange']

dict keys are unique, and Python dictionaries preserve insertion order. Each repeated key is ignored, so the first appearance of each value remains.

For a reusable and explicit approach, write a function:

def remove_duplicates(items):
    seen = set()
    unique_items = []

    for item in items:
         item   seen:
            seen.add(item)
            unique_items.append(item)

     unique_items

(remove_duplicates([, , , , ]))

Consider this function call:

def remove_duplicates(items):
    seen = set()
    result = []

    for item in items:
        if item not in seen:
            seen.add(item)
            result.append(item)

    return result

print(remove_duplicates(["red", "blue", "red", "green"]))

Execution trace:

1. seen starts as {} and result starts as [].
2. Item is "red". It is not in seen, so add it.
   • seen becomes {'red'}
   • result becomes ['red']
3. Item is "blue". It is new, so add it.

Duplicate removal is common whenever data comes from users, files, APIs, or databases.

• User-entered tags: A post with ['python', 'api', 'python'] should show each tag once.
• Email recipients: Remove repeated addresses before sending a notification.
• API results: Combine records from several endpoints without processing the same ID twice.
• File processing: Avoid importing the same filename multiple times.
• Permissions: Build a list of unique roles assigned through several groups.
• Data cleanup: Remove repeated product codes from a spreadsheet import.

In real projects, developers often choose the method based on whether order matters and whether they need extra information.

Validate input early

def validate_categories(categories):
    if len(categories) != len(set(categories)):
        raise ValueError("Categories must not contain duplicates.")

This rejects invalid input instead of silently changing it.

Normalize data before deduplicating

Values that look equivalent may not compare as equal until they are cleaned:

def unique_emails(emails):
    normalized = [email.strip().lower() for email in emails]
    return list(dict.fromkeys(normalized))

This treats " Ada@Example.com " and "ada@example.com" as the same email.

Deduplicate objects by an ID

For dictionaries or objects, developers commonly track a specific key:

def ():
    seen_ids = ()
    result = []

     user  users:
         user[]   seen_ids:
            seen_ids.add(user[])
            result.append(user)

     result

Using set when output order matters

items = ["first", "second", "first"]
unique_items = list(set(items))

A set does not represent an ordered sequence. Do not rely on this conversion to keep the original order. Use list(dict.fromkeys(items)) or the seen-set loop instead.

Forgetting that set() needs hashable values

Lists and dictionaries cannot be inserted into a set:

items = [[1, 2], [1, 2]]
set(items)  # TypeError: unhashable type: 'list'

For nested lists, compare a hashable representation such as a tuple:

items = [[1, 2], [1, 2], [3, 4]]
unique_items = [list(item) for item in dict.fromkeys(map(, items))]

A list can contain duplicates and preserves positions. A set stores unique hashable values and is optimized for membership checks. A dictionary stores key-value pairs, but its unique ordered keys make useful for deduplication.

# Check for any duplicates
has_duplicates = len(items) != len(set(items))

# Remove duplicates; order does not matter
unique_items = list(set(items))

# Remove duplicates; preserve first-occurrence order
unique_items = list(dict.fromkeys(items))

# Explicit, customizable order-preserving approach
seen = set()
unique_items = []
for item in items:
    if item not in seen:
        seen.add(item)
        unique_items.append(item)

• set and dictionary keys require hashable values.
• Use normalization first when comparison should ignore spaces or letter case.
• For dictionaries or objects, deduplicate using a stable key such as item['id'].
• Keep the original list unchanged unless the task specifically requires mutation.

How do I know whether a Python list has duplicates?

Compare the list length with the length of a set:

has_duplicates = len(items) != len(set(items))

What is the shortest way to remove duplicates but keep order in Python?

Use:

unique_items = list(dict.fromkeys(items))

Does list(set(my_list)) preserve list order?

No. A set is not an ordered representation of the original sequence, so use an order-preserving approach when order matters.

Does removing duplicates change the original list?

Not when you assign the result to a new variable:

unique_items = list(dict.fromkeys(items))

items remains unchanged.

Can I remove duplicate dictionaries from a list with set()?

No. Dictionaries are unhashable. Instead, use a field such as an ID as the value stored in seen.

Which duplicate is kept when preserving order?

The first occurrence is kept. Later occurrences are skipped.

• Python lists — Lists store ordered collections and may contain repeated values.
• Python sets — Sets store unique hashable values and support fast membership checks.
• Python dictionaries — Dictionary keys are unique and preserve insertion order, enabling dict.fromkeys().
• Membership operators (in and not in) — Used to test whether a value has already been seen.
• List comprehensions — Useful for normalizing or transforming data before deduplication.
• Data validation — Duplicate detection is often part of validating user input and API payloads.
• Hashable objects — Explains why strings and tuples work in sets while lists and dictionaries do not.