Menu

1 / 2

Rate-Limited Job Queues with Fixed Concurrency, Batch Publishing, Retry, and Idempotency

DEV Community: go·RhettFletcher9678·25 days ago
#8KSFFyr7
#devtogo#devto#dev#context#queue
Reading 0:00
15s threshold

For rate-limited job processing, a queue and a fixed-concurrency worker are safer than adding another cron run. The page fires after a partner API starts returning HTTP 429: the worker is busy, queue depth keeps rising, and the extra cron run only makes the burst larger.

Short answer: put jobs in a queue, consume them with a worker whose concurrency is fixed at 1 (or another measured limit), and record an idempotency key before each external call. Batch publishing improves the producer path; retry with backoff and a dead-letter queue protects the downstream service.

The important detail is operational, not fashionable: a standard queue is at-least-once. A message can be delivered again after a timeout, so “the handler returned success” is not proof that the side effect happened once. Treat every delivery as a possible duplicate.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.