Yesterday we finally traced why 96000 model requests a day were costing more than the traffic suggested. About 6720 requests, or 7%, cross a 5 second client timeout. The original model request keeps running while 2 different middleware layers each launch their own fallback. Those timed out customer requests turn into over 20k billable model calls which leaves us paying for 13k duplicate calls every day. At an average 9000 tokens per call, the duplicate branches add nearly 121 million tokens daily.

The first fix was lowering the retry count. It barely moved anything because both middleware layers still believed they owned retry behavior. One layer handled the API boundary and the other wrapped the model client so each saw the timeout independently and started its own fallback. Cancellation propagation never reached the original provider request, either.…