Originally published on tamiz.pro .
At 3:04 a.m. on a Tuesday, your self-hosted LLM stack stops answering. Not with a 500, not with a clean timeout — it just… stops. The pager fires. You SSH in and find a process chewing 100% CPU, a GPU sitting at 0% utilization, and a log file full of ECONNRESET errors. The agent fleet that was supposed to be idle is actually mid-retry-storm. The MoE model you offloaded to CPU RAM has thrashed its way into a swap loop. And the fetch() call that triggered it all — the one that has been hanging for 47 minutes — never had a timeout in the first place.
This is the local LLM production trap. It isn't one bug. It's three independent design decisions that are each reasonable in isolation, but which compound into a cascade failure at the worst possible moment. Let's dissect all three, see exactly how they interact, and then harden the stack against them.
1. The Promise of Local LLMs in Production
2. MoE Offloading: The Expert That Isn't There
3.…