There is a proverb I grew up with: don’t trouble the trouble, otherwise trouble will trouble you. It’s usually offered as advice about people. It’s even better advice for dealing with distributed systems.
Every engineer shares the same instinct. A call fails. Try it again. It’s such a reasonable response that most frameworks now handle it for you silently, baking in a sensible-looking default of three attempts. You don’t write the retry; you inherit it. For a single client, that’s correct. Networks drop packets. Instances get recycled mid-flight. Trying again is often all it takes. The trouble starts when there are ten thousand clients, and they all have that exact same instinct at the exact same moment.
The Arithmetic Nobody Does
Imagine a service that comfortably handles 1,000 requests per second with healthy headroom. It can manage about 1,200 before it starts queueing.
Now, something downstream degrades.…