Menu

I ran 33 LLM models behind one endpoint for a month. Here is what broke.
📰

I ran 33 LLM models behind one endpoint for a month. Here is what broke.

DEV Community: node·Lv Shao Mu·9 days ago
#3qLd9JTR
#devtonodejs#devto#dev#model#break
Reading 0:00
15s threshold

I built a small gateway that pools free LLM tiers — NVIDIA NIM, Kilo, OpenRouter, a couple of

community endpoints — behind a single OpenAI-compatible URL. Thirty-three models, one API key,

no monthly bill. It ran real daily work for a month.

It also broke in six different ways, and every single break was something I would have avoided

if someone had written it down first. So I did.

The numbers first

What Measured

Free tier rate limit 40 requests/min, sliding window

Burst probe result 45 requests → 24 success, 20 rate-limited, 1 upstream 503, in 12.8s

Fast model, 95K context 18.6s

Slow model, same context 83s

Requests that hung and then failed every one, at 178–194s

That last row is the one that cost me the most sleep.

Break 1: the 180-second hang that was not my bug

Every request through one particular relay hung for exactly 178 to 194 seconds and then failed

with a connection error. My code was fine.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.