Anyone running an LLM feature for more than a few months has a fallback path: a second provider, a router, or a forty-line wrapper that tries the next model when the first one returns 529 or 503 . The design work usually stops at "if the primary fails, use the secondary." That is the easy half.
The hard half is what happens to the secondary at the exact moment the primary trips.
Failover is a synchronised event
When the primary starts returning overload errors, every client fails over within the same few seconds. The secondary does not receive a trickle; it receives its normal traffic plus the entire primary's traffic, arriving as one burst, with retry loops attached.
If the secondary was sized for its own load plus a small margin, it was sized for a number that never occurs during an incident.…