Press enter or click to view image in full size

Kafka Exactly-Once Semantics: The Payment Bug Our Staff Engineer Almost Missed

Our Kafka transactions were working perfectly. The payment provider was processing requests correctly. Somehow, we still managed to charge the same customer twice.

Devrim Ozcay | Founder, ProdRescue

10 min read15 hours ago

--

The first duplicate payment arrived at 10:42 on a Thursday morning, which was inconvenient because our engineering team had spent the previous Wednesday celebrating the successful rollout of Kafka exactly-once processing. We’d upgraded the consumers, enabled transactions, reviewed the producer configuration, and added enough integration tests to make the CI pipeline reconsider its career choices.

Everything looked good. Kafka transactions were committing successfully, consumer offsets were advancing as expected, and our payment service had been running without significant errors.…