Merged two email lists and now half the addresses appear twice. Exported keywords from three tools and the same terms repeat on every page. Pasted a log file and the same error lines show up forty times.
Duplicate lines are the most boring data-cleaning job there is β and the one most likely to go wrong, because the obvious fixes quietly destroy something else.
The problem with the obvious fixes
Excel's "Remove Duplicates" works, but it reorders nothing and tells you nothing β and if your list has 50,000 rows, Excel is a slow way to do a 5-second job. It also treats Delhi and delhi as different entries, and a trailing space makes two "identical" lines different.
sort -u in the terminal is fast and free, but it sorts. If your keyword list is in priority order, or your log is in time order, sorting destroys the information you actually care about. The output is clean and useless in the wrong order.
Python's set() has the same flaw β a set has no order at all.β¦