The backup job had been green for fourteen months. Every night at 02:00 it ran mysqldump , gzipped the output, pushed it to object storage and posted a tick to the ops channel. Then a developer ran a migration against the wrong connection and dropped a tenant table. We pulled last night's dump, started the restore, and watched it fail at 61%: the dump had been taken without --single-transaction on a set of InnoDB tables that were being written during the run, and a later change had added a view referencing a table that was dumped after it. Every backup for months had the same problem. The job never failed because writing a broken file is still a successful write.

I wrote this with help from AI tools and checked every command, script and config myself on MySQL 8 and PostgreSQL 16.…