Description

When one service does the work and another records it, some work runs twice. Idempotency is a correctness requirement.

This post covers two writes that look like one step: doing some work, and recording that it was done.

In this design the work runs in one service. Another service writes the record. No transaction spans both. So some work will run twice. The shape of the design guarantees it, and high load is not required. That makes idempotency a correctness requirement. If a stage cannot be made idempotent, the architecture has to change. Tighter timeouts will not fix it.

The architecture

The platform is an enterprise quoting system. It runs as Spring Boot services on EKS, and around 30 of them share one self-managed Oracle 12c instance.

Each in-flight workflow has one row in a PENDING_WORK table. The row is keyed by GROUP_ID and QUOTE_ID and carries a STAGE and a STAGE_STATE . A scheduler runs on every pod, and ShedLock makes it a singleton. It orchestrates and holds no business logic.…