Pulling an open-source model, starting a server, and getting an answer to your prompt should take less than 30 minutes. In half an hour or so, you’ve got yourself a prompt-ready machine. But, send 10k+ requests a day to the same inference server, and you won’t be so pleased with its performance.

The difference between demo and production-ready inference is wild.

Let’s see the main options out there for running (open-source) models, be it 10 requests a day or 10,000.

TL;DR Decide what “production” means for your workload first, then pick the server that covers most of your needs out of the box. One large generative model calls for a fast runtime, whereas the small models around your agent (embedding, reranking, extraction) need a multi-model server.

Want the broader roundup, including Ollama and LocalAI?…