Some work doesn't fit in a request. Moving a customer's data to new infrastructure. Onboarding a merchant across five systems. Generating, approving and sending a large batch of documents. These jobs have many steps, take minutes to days, call systems that fail, and must survive deploys and crashes.
Most teams start the same way: a queue, a cron job and a status column.
What queue + cron becomes
Version one is clean: a job row with status = PENDING , a worker that picks it up, does the work, sets DONE . Then reality adds requirements, one at a time:
A step fails transiently β add a retries column and a RETRYING status.
A worker dies mid-job β add locked_by and locked_at , and a cron that releases stale locks.
Step 4 must not run twice β add per-step status columns.
A step needs to wait for a human approval or a 24-hour cool-down β add resume_after and another cron.
Step 6 fails permanently after steps 1β5 succeeded β write undo logic, scattered across the codebase.
Nobody can tell where a job is ββ¦