My Retry Wrote the Row Twice, and the Idempotency Key Was the Bug
Here is a failure shape I keep meeting in batch jobs: one input, two writes to the database, and a clean 200 on every log line I check. Which attempt actually did the work? Did the queue hand me the same job twice? Did my own retry fire while the first call was still in flight, so two workers raced to the same finish line?
None of those questions can be answered from timing alone, because a duplicate looks exactly like a success. So for 48 hours I stopped adding retries and started counting duplicates instead. I worked it in three passes: where a duplicate is born, how to fingerprint a unit of work, and how to prove the guard holds when I inject faults. These are the field notes from that pass, including the parts that broke.
Pass 1: Three places a duplicate is born
Before writing any code I listed the birthplaces, because each one needs a different fix.…