Menu

Your Retry Logic Might Be the Reason Your System Won't Recover
📰

Your Retry Logic Might Be the Reason Your System Won't Recover

#hackernoon#technology#retry#service#system
Reading 0:00
15s threshold

There is a proverb I grew up with: don’t trouble the trouble, otherwise trouble will trouble you. It’s usually offered as advice about people. It’s even better advice for dealing with distributed systems.

Every engineer shares the same instinct. A call fails. Try it again. It’s such a reasonable response that most frameworks now handle it for you silently, baking in a sensible-looking default of three attempts. You don’t write the retry; you inherit it. For a single client, that’s correct. Networks drop packets. Instances get recycled mid-flight. Trying again is often all it takes. The trouble starts when there are ten thousand clients, and they all have that exact same instinct at the exact same moment.

The Arithmetic Nobody Does

Imagine a service that comfortably handles 1,000 requests per second with healthy headroom. It can manage about 1,200 before it starts queueing.

Now, something downstream degrades.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.