Notes from a 2 AM On-Call: Why Our Bifrost Gateway Silently Dropped 40% of Streaming Requests

At 2:14 AM last Tuesday, our on-call dashboard lit up. The symptom was bizarre: SSE streaming responses from our maximhq/bifrost gateway were working perfectly for the first 5-10 concurrent requests, then silently failing for everything beyond that threshold. No 502s, no connection resets, no OOM kills in the logs. Just... silence. The client SDKs sat there waiting for chunks that never arrived.

The Failure Symptom

Our stack: maximhq/bifrost (v0.8.x) running behind nginx as a reverse proxy, routing to multiple LLM providers with streaming enabled. When concurrent streaming requests hit ~12-15 simultaneous connections, new clients would establish WebSocket handshakes successfully, receive the initial SSE headers, then get zero data chunks. The bifrost process showed no errors.…