Menu

1 / 4

My server didn't go down. Chasing the false alarm hardened everything else.

DEV Community: dockerΒ·Luis FariaΒ·about 1 hour ago
#DDJHKBJ3
#devtodocker#devto#dev#sentry#test
Reading 0:00
15s threshold

On November 7, Discord pinged me: πŸ”΄ Container(s) Down: backend_api. I opened my site on my phone and it didn't load. That looked like proof.

It wasn't. I SSHed into the droplet and read the logs. Every container had been up the whole time. My phone's 4G had dropped out, and the alert was false.

That should have been a five-minute investigation, but it turned into two days and 13 pull requests, because every fix exposed the next thing nobody had looked at. This post is the trail: what was wrong, how each fix was checked on the live server, and the surprises, which taught me more than the plan did.

Contents

The setup

13 PRs at a glance

1. The alert that cried wolf

2. What the logs said

3. The config that only lived on the server

4. The header I sent twice

5. Rate limits, and the IP everyone shares

6. The upgrade that wasn't

7.…

Continue reading β€” create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.