Menu

The Job Died at Noon, Restarted at 3pm — and Processed the Whole Morning Twice
📰

The Job Died at Noon, Restarted at 3pm — and Processed the Whole Morning Twice

DEV Community: database·lujianfu2008·8 days ago
#8CEoULIW
Reading 0:00
15s threshold

A true story, and one of the most expensive kind — because the code looked fine.

A scheduled batch job ran at noon and failed. The on-call engineer traced it to a flaky downstream API, and at 3pm restarted the job. It ran clean. Everyone went back to work.

Then the complaints started coming in.

The restart had re-processed the entire morning batch from the beginning : orders that were already placed were placed again, charges already taken were taken again, notifications already sent went out again. Customers saw duplicate billing.

This wasn't sloppy code. The design was the textbook scan table → process each row → mark status . It had been in production for two years without incident. It broke under one specific failure mode: the job didn't fail to finish — it got about halfway, then the process ceased to exist.

Those two things look completely different in code.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.