Menu

How to Remove Duplicate Lines from a List (Free, Keeps Order, No Uploads)
πŸ“°

How to Remove Duplicate Lines from a List (Free, Keeps Order, No Uploads)

DEV Community: beginnersΒ·Sujal KhatriΒ·about 19 hours ago
#NL5YiDy7
#devtobeginners#devto#dev#example#lines
Reading 0:00
15s threshold

Merged two email lists and now half the addresses appear twice. Exported keywords from three tools and the same terms repeat on every page. Pasted a log file and the same error lines show up forty times.

Duplicate lines are the most boring data-cleaning job there is β€” and the one most likely to go wrong, because the obvious fixes quietly destroy something else.

The problem with the obvious fixes

Excel's "Remove Duplicates" works, but it reorders nothing and tells you nothing β€” and if your list has 50,000 rows, Excel is a slow way to do a 5-second job. It also treats Delhi and delhi as different entries, and a trailing space makes two "identical" lines different.

sort -u in the terminal is fast and free, but it sorts. If your keyword list is in priority order, or your log is in time order, sorting destroys the information you actually care about. The output is clean and useless in the wrong order.

Python's set() has the same flaw β€” a set has no order at all.…

Continue reading β€” create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.