On October 20, right after a deploy, I ran our queue-count command. The registry of started jobs in our main queue listed 27 jobs. No worker was running any of them. Every one had been killed by a deploy.

The next morning, a worker died while cleaning up after four jobs killed by that morning's deploy. It was idle, so no job was lost, and the pool started a new one. Following that crash led back to a line we thought we had fixed in May.

Some context. I run engineering at GoodBarber , an app platform. Our AI app builder writes the code of an app section from a description, and each generation runs as a job on a pool of RQ workers: rq 2.12, started with django-rq's rqworker-pool , with the tasks declared through django-tasks. It all runs in Docker Compose on two hosts, and a deploy takes the containers down, then brings the new image up. This post is about what those deploys were doing to the workers and the four fixes we shipped on October 22.…