Menu

Celery visibility_timeout Ran My Claude Jobs 3.3x: 1,800 Became 5,913
📰

Celery visibility_timeout Ran My Claude Jobs 3.3x: 1,800 Became 5,913

DEV Community: python·jidonglab·1 day ago
#MWhaCYF2
#devtopython#devto#dev#task#tasks
Reading 0:00
15s threshold

My Postgres table said 1,800. My Claude usage graph said I had paid for something closer to 6,000.

Both numbers were correct, and the database was the one hiding the problem. Every night I summarize 1,800 support tickets with Claude through a Celery queue on Redis. The summaries were fine, there was exactly one row per ticket, and nothing errored. But the Celery visibility_timeout on my Redis broker was quietly re-running most of those tasks, some of them six times, and my upsert erased the evidence every time.

This is the autopsy.

TL;DR

On Redis (and SQS), Celery redelivers any message that isn't acknowledged within visibility_timeout , which defaults to 1 hour .

Tasks scheduled with countdown or eta sit in a worker's memory unacknowledged until they run. If the wait is longer than an hour, the message goes back on the queue and another worker grabs a second copy. Then a third. It repeats every hour.

I spread 1,800 tasks over 6 hours with countdown=i * 12 .…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.