Our MySQL server was perfectly healthy, and the site was still down for 8 minutes.
No deploy, no traffic spike, no slow query. The app server just could not reach the database for a while, and Laravel waited. And waited. Turns out a Laravel MySQL connection sets no connect timeout of its own, and when the connect fails, Laravel tries again, three more times.
What Happened
The project I work on runs on Hetzner. The app box talks to MySQL over Hetzner's private network. One afternoon that internal network failed in one datacenter. MySQL was fine. The app box was fine. They just could not see each other.
The Laravel log filled with this:
SQLSTATE[HY000] [2002] Connection timed out
Enter fullscreen mode
Exit fullscreen mode
An error in the log is normal during a network problem. But the whole site stopped answering, including routes that never touch the database.
The box has one PHP-FPM pool with 5 workers, and two Laravel apps share it: the storefront API and the API our ERP uses for its data exchange.…