Our WebRTC signaling server was unavailable for 2 days, 17 hours and 55 minutes, from 2026-11-08 13:46 to 2026-11-11 07:41 UTC. The last commit to its code was on 2026-03-23 and the Deployment had pointed at the same image tag since then. The outage started with a disk filling up on the node. It lasted because of what a registry cron job had done to that image on 2026-03-24.
This is a two-node k3s cluster that I run alone for a few side projects. Times are UTC.
What it looked like
By the morning of 2026-11-11 the Deployment's ReplicaSet had three pods. The original, created in March, had ended on 2026-11-08 at 13:46:42 with exit code 0 and phase Succeeded . A replacement created one second later had never run and ended with ContainerStatusUnknown , exit code 137. A second replacement, created at 16:10:52, was Pending with this event repeating every five minutes:
Failed to pull image "registry.example.com/dam-signaling:prod-<sha>": failed to pull and unpack image ...: .../manifests/sha256:211614369c9d...…