We deduplicated webhooks the obvious way. Handler receives an event, runs SELECT on processed_events by event_id, and if nothing is found, does the work and inserts the row at the end.

It held for months. Then a customer got two receipt emails and a loyalty-points credit twice for the same payment.

The cause: the gateway redelivered the event 40 ms after the first attempt, because our first response was slow (a cold DB connection pushed it to about 3 seconds, and their timeout was 2). Both deliveries landed on different pods. Both ran the SELECT before either had inserted. Both saw an empty table, and both did the work.

In logs it looked like duplicate delivery, but the gateway was behaving correctly. Our check-then-act was the bug.

The fix was to flip the order. Insert the event_id first, into a table with a unique constraint, inside the same transaction as the side effects. If the insert hits a conflict, return 200 and stop.…