How a preStop hook, one Unix signal, and the right grace period stopped Kubernetes from killing our long-running Sidekiq jobs in the middle of their work.

Every deployment looked clean. Green pipeline, new pods up, old pods gone. And every deploy, a handful of background jobs quietly died halfway through.

We ran a big part of our product on Sidekiq workers in Kubernetes. Short jobs were fine. The long ones, the jobs that needed a few minutes to finish, were the problem. If a deploy landed while one of them was running, it got cut off. Sometimes Sidekiq retried it, and it finished later. Sometimes it left work half done, and someone had to clean up by hand.

Nothing was "broken." Kubernetes and Sidekiq were both doing exactly what they were told.…