I built a small gateway that pools free LLM tiers — NVIDIA NIM, Kilo, OpenRouter, a couple of
community endpoints — behind a single OpenAI-compatible URL. Thirty-three models, one API key,
no monthly bill. It ran real daily work for a month.
It also broke in six different ways, and every single break was something I would have avoided
if someone had written it down first. So I did.
The numbers first
What Measured
Free tier rate limit 40 requests/min, sliding window
Burst probe result 45 requests → 24 success, 20 rate-limited, 1 upstream 503, in 12.8s
Fast model, 95K context 18.6s
Slow model, same context 83s
Requests that hung and then failed every one, at 178–194s
That last row is the one that cost me the most sleep.
Break 1: the 180-second hang that was not my bug
Every request through one particular relay hung for exactly 178 to 194 seconds and then failed
with a connection error. My code was fine.…