On November 7, Discord pinged me: π΄ Container(s) Down: backend_api. I opened my site on my phone and it didn't load. That looked like proof.
It wasn't. I SSHed into the droplet and read the logs. Every container had been up the whole time. My phone's 4G had dropped out, and the alert was false.
That should have been a five-minute investigation, but it turned into two days and 13 pull requests, because every fix exposed the next thing nobody had looked at. This post is the trail: what was wrong, how each fix was checked on the live server, and the surprises, which taught me more than the plan did.
Contents
The setup
13 PRs at a glance
1. The alert that cried wolf
2. What the logs said
3. The config that only lived on the server
4. The header I sent twice
5. Rate limits, and the IP everyone shares
6. The upgrade that wasn't
7.β¦