The disk that filled itself

TL;DR : my homelab box hit 100 percent disk full out of nowhere. I deleted half the things I could find, df still said full, du said I had plenty of space. Turned out the disk was holding on to files I had already deleted, because a long-running process still had them open. lsof +L1 was the magic. A service restart was the fix.

So there I was, on a perfectly normal evening, ssh'd into the homelab box because something had stopped responding. The first thing I check on any "why is this dying" run is df -h , almost as a reflex.

Filesystem Size Used Avail Use% Mounted on /dev/nvme0n1p2 450G 448G 2G 100% /

Enter fullscreen mode

Exit fullscreen mode

Cool. So that is why nothing is working.

I have a deal with this box. It runs my self-hosted things, it does not ask for much, and once a quarter or so I prune some old container images and we move on.…