As a medium-sized company with a large number of monthly active users (MAU) and a few digital products, you will have realised by now that using free ChatGPT is not enough. Yes, you have learned about skills.md and the expensive Claude subscription. But is that enough? In this article, I will share the knowledge needed to design a multi-agent LLM system for mid-scale organisations.

The golden rule of working with LLMs: treat publicly available models as a brain, not as eyes and ears. An LLM doesn't actually know anything — it generates something that looks like an answer. That's why we, as developers, need to add plenty of validation along the way from the user's request to the model's response. Keep one simple rule in mind: if a human could mess it up, an LLM definitely will, because LLMs have no common sense.

The general pipeline for getting an answer from an LLM looks like this: indexing > retrieval > generation.…