Menu

Five LLM gateway bugs that look like success (and how to test for them)
📰

Five LLM gateway bugs that look like success (and how to test for them)

DEV Community: opensource·ArkNet Digital·2 days ago
#sGp1rSrj
#devtoopensource#devto#dev#gateway#usage
Reading 0:00
15s threshold

If your app talks to models through an OpenAI-compatible gateway (LiteLLM or one of its many peers), that gateway is doing a lot on every request. It re-streams the response chunk by chunk, recomputes token usage, works out what the request cost, and has to pass upstream failures back to you honestly.

That is stateful, timing-sensitive code sitting in the hot path of every call. The bugs it produces share one nasty property: they look like success.

The five failure modes

1. A dropped or reordered stream chunk. The answer arrives one word short. Nobody notices in a manual test, because you'd have to know what the full answer should have been.

2. Wrong token usage. prompt_tokens , completion_tokens or total_tokens is off, either in the final streamed chunk or the non-streamed body. Your usage dashboards are now fiction.

3. Wrong cost. The gateway still returns a number. You only find out it was wrong when the bill doesn't match the usage.

4. Cross-stream leakage.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.