The Incident: When a Routine Backup Fills the Disk

In DevOps, some of the most challenging incidents begin with something that looks simple.

One day, while working on our Kubernetes infrastructure, I investigated a master node whose root filesystem had reached 100% disk usage.

At first, it looked like a typical disk-space problem. Maybe logs had grown too large, container images were consuming storage, or some application was generating unexpected data.

But after investigating the filesystem, I discovered that the problem was related to something we depend on for disaster recovery: etcd backups.

Our backup process was generating large snapshots, but multiple etcdctl snapshot save processes had become stuck.…