Introduction

We hit a scaling cliff when our product moved from a few thousand concurrent users to tens of thousands. The thing that looked trivial in staging — pushing events over WebSockets and orchestrating AI agents — started manifesting as tail latency spikes, connection storms, and a surprising amount of bookkeeping code in our app layer.

Here’s what we learned the hard way building a realtime, event-driven backend for AI workflows and multi-tenant SaaS.

The Trigger

The immediate trigger was simple: a big customer started running thousands of long-running inference sessions using multiple agents that exchanged messages in realtime.

At first, this looked fine — we had a single message broker and a WebSocket cluster.…