Remove Duplicate Lines From a 48,000-Line CSV Without Losing the Original Order

Last month I had to merge two exports from our billing system into one import file. The first export had 31,207 customer email addresses, the second had 17,706, and a few thousand lines overlapped. The CRM import endpoint rejects the whole file if it contains a single repeated email, so deduplication had to happen before upload.

There was a second constraint that tripped me up: row order drove import priority. Whichever line appeared first had to stay. So sort -u , the command half of my brain reached for, was off the table from the start. Here are three approaches that do work, tested on Ubuntu 22.04 with Python 3.11.1 and GNU awk 5.1.1.

Method 1: Python, streaming with a set

The obvious one-liner is list(set(lines)) . I've written it myself under deadline pressure. Don't.…