Menu

A job queue in Postgres, two months later
📰

A job queue in Postgres, two months later

DEV Community: go·Rowan Feld·3 days ago
#N2BqaL08
#devtogo#devto#dev#queue#skip
Reading 0:00
15s threshold

Rowan Feld

We moved a small job queue from Redis onto Postgres two months ago, mostly to get transactions and a single backup story. Here's what actually changed.

Throughput was never the problem. We run about 40 jobs a second at peak, and SELECT ... FOR UPDATE SKIP LOCKED handles that without breaking a sweat. The counter arguments about Postgres queues are usually about ten thousand a second, which we will never see.

What did need thought was stuck jobs. A worker that dies mid-job leaves a locked row, and SKIP LOCKED will skip it forever. We added a locked_until column and a reaper that clears expired locks every thirty seconds. That reaper is the actual queue implementation; the select is easy.

The other surprise was vacuum. A table that churns through rows fast needs its autovacuum settings tuned, or the table bloats and the queue gets slower for no obvious reason.

Two months in: fewer moving parts, one backup, and a reaper job I check on Mondays. I'd make the same trade again.