How multi-provider LLM routers silently fail

A failure mode common to several Python LLM routers: a 429 caused by an

exhausted long-period quota is treated identically to a 429 caused by a

transient per-minute rate limit. The cooldown TTL ends up applied to

both, and one of the two cases is wrong by orders of magnitude.

This essay describes the failure mode in concrete terms and outlines the

small fix.

The shape of the failure

Most LLM routers track a single "this provider is unhealthy until X"

field per deployment. When a request fails, the router sets X = now +

cooldown , and is_call_allowed() returns False until then.

That works perfectly for a transient per-minute rate limit, where

"cooldown" is correctly measured in seconds.…