Latency is the wrong metric for fallback routes

I built a fallback chain for an agent that searches the web, pulls a few pages, and then writes an answer. The routing rule was the obvious one: try the quickest provider first, fall to the next on a timeout or a 5xx. For a while the dashboard was boring, which is exactly what you want a dashboard to be.

The failure did not look like a failure. Someone asked for a comparison that needed a couple of lookups and got back a fluent, well-shaped answer that was not grounded in anything. No search had happened. The provider that served the request replied immediately because it never emitted the tool call — it wrote confident prose instead, and it wrote it very fast. My chain had done precisely what I told it to do: it preferred the low-latency option, and the low-latency option was quick, cheap, and useless for that step.

Losing a hop is cheap.…