Originally published at tothenoc.substack.com
Why every ToTheNOC image is launched on a real instance before it is listed.
We build hardened Linux images. AlmaLinux, Rocky Linux and Debian, x86 and ARM.
In October one of our AlmaLinux 9 builds passed the hardening scan. Clean. Then we launched it on a real instance and it never came up. No SSH. The serial console said "Read-only file system", and after that cloud-init, auditd and sshd all failed.
The cause was one line. The hardening profile wants /tmp on its own mount, so the recipe added a tmpfs line in fstab. On the AlmaLinux base image, tmp.mount is masked. systemd puts that masked unit in the boot transaction, the branch falls out, the root filesystem is never remounted read-write. Everything that needs to write to disk dies, sshd included.
Nothing in the build told us. The scan checks files. It does not boot the system.
A few weeks later, same story on ARM.…