If your app talks to models through an OpenAI-compatible gateway (LiteLLM or one of its many peers), that gateway is doing a lot on every request. It re-streams the response chunk by chunk, recomputes token usage, works out what the request cost, and has to pass upstream failures back to you honestly.
That is stateful, timing-sensitive code sitting in the hot path of every call. The bugs it produces share one nasty property: they look like success.
The five failure modes
1. A dropped or reordered stream chunk. The answer arrives one word short. Nobody notices in a manual test, because you'd have to know what the full answer should have been.
2. Wrong token usage. prompt_tokens , completion_tokens or total_tokens is off, either in the final streamed chunk or the non-streamed body. Your usage dashboards are now fiction.
3. Wrong cost. The gateway still returns a number. You only find out it was wrong when the bill doesn't match the usage.
4. Cross-stream leakage.…