On a Sunday night in October the database behind our returns service shut itself down with a PANIC in its log: it could not write to a file in pg_wal because there was no space left on the device. Nobody was paged. Our disk alert pages at ninety seven percent full, and the dashboard for that volume, when we opened it on Monday morning, said ninety five.
The volume is two terabytes of ext4, and for weeks it had been filling slowly with a table of return photographs nobody had archived. The warning at ninety percent had raised a ticket on the Tuesday, and the ticket was planned for the following week, because at that rate the page would arrive with days to spare.
Both numbers on our dashboards came from node_exporter, and both were correct. It exports two figures for free space. node_filesystem_free_bytes counts every free block. node_filesystem_avail_bytes counts the free blocks an ordinary user is allowed to write to.…