Menu

1 / 2

ImagePullBackOff on an image nobody had touched since March

DEV Community: kubernetes·KYUNGHWAN KIM·5 days ago
#r77BfjZ5
#devtokubernetes#devto#dev#manifest#registry
Reading 0:00
15s threshold

Our WebRTC signaling server was unavailable for 2 days, 17 hours and 55 minutes, from 2026-11-08 13:46 to 2026-11-11 07:41 UTC. The last commit to its code was on 2026-03-23 and the Deployment had pointed at the same image tag since then. The outage started with a disk filling up on the node. It lasted because of what a registry cron job had done to that image on 2026-03-24.

This is a two-node k3s cluster that I run alone for a few side projects. Times are UTC.

What it looked like

By the morning of 2026-11-11 the Deployment's ReplicaSet had three pods. The original, created in March, had ended on 2026-11-08 at 13:46:42 with exit code 0 and phase Succeeded . A replacement created one second later had never run and ended with ContainerStatusUnknown , exit code 137. A second replacement, created at 16:10:52, was Pending with this event repeating every five minutes:

Failed to pull image "registry.example.com/dam-signaling:prod-<sha>": failed to pull and unpack image ...: .../manifests/sha256:211614369c9d...…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.