Retries are the cheapest reliability feature there is: a request fails because of a dropped packet or a pod restart, you try again, it works. Most transient failures are gone in milliseconds.

They're also one of the most common ways a small incident becomes a large one. A database hiccups for a minute. Every caller retries. The callers of those callers retry too. The database — the one thing that needed less traffic to recover — now receives several times its normal load, and the one-minute blip becomes a long outage. This pattern is often called a retry storm .

Here are the four tools that keep retries on your side.

The arithmetic of retry amplification

Imagine a request that goes through three services before it reaches a database, and each service retries a failed call 3 times in total (1 attempt + 2 retries).

Every layer that retries multiplies the load on the layers below. Explore the interactive diagram

When the database is failing, one user request becomes 3 × 3 × 3 = 27 database calls .…