We found a pretty nasty issue in our webhook pipeline a few months ago.

A merchant reached out saying a customer had successfully paid for a yearly subscription but never got access to the product. No email, no provisioning, nothing.

At first glance everything looked normal. The payment existed in the database, status was marked successful, charge ID was there, fully committed.

But the webhook event tied to it just, didn’t exist lol. Not failed. Not queued. Not retrying. Just gone.

Took a while to figure out what was happening, but eventually we traced it back to deploys. A pod would receive the request, start processing it, then get terminated mid-flight during rollout.…