14:07 UTC. PagerDuty fires on fulfillment-queue-depth , 340% of baseline and climbing. Checkout's on-call pulls up the dashboard expecting a stuck consumer or a slow downstream call. Instead she finds the queue isn't slow, it's just full of orders nobody placed twice on purpose.

the scramble

First theory: the mobile app's retry logic, rewritten three weeks ago to handle flaky connections more aggressively, is double-submitting. Someone pulls the client-side request logs for a sample of the duplicate order IDs. Each one shows exactly one tap, one request, one Idempotency-Key header generated client-side and reused correctly across any retries the client itself issued. The client is behaving. Ruled out in under five minutes.

Second theory, and the one that eats fifteen minutes: API Gateway retrying a Lambda invocation that looked like it timed out, but actually succeeded after the gateway gave up and resent the request. This would produce exactly this symptom, two invocations for one logical request.…