A one-line dedupe rule was quietly deleting one in six real job postings from my dataset. Here is how it showed up, and the field I forgot to put in the key.

The symptom was a number I was proud of

After cleaning up a date bug, I looked at my pipeline's own funnel:

18,833 postings scraped → 15,548 after dedupe

Enter fullscreen mode

Exit fullscreen mode

A 17.4% duplicate rate. I had been treating that as proof the sources were messy and my dedupe was doing heroic work. It is not. On real job-board data, cross-source duplicates sit around 1–3%. Anything much above that is usually not the data being dirty — it is your key being too coarse.

So I stopped celebrating and measured what was actually colliding.

Measuring the collision

I took seven companies ( stripe , gitlab , anthropic , figma , brex , ramp , openai ), 2,835 postings, grouped them by my dedupe key, and then asked a much more useful question: inside each collision group, how many distinct locations are there?

The key was:

key = ( job [ "…