Most guides to self-hosted AI stop at ollama run llama3 . You get the model running, you have a terminal prompt, and the guide says congratulations. Then you try to use it seriously — share it with a teammate, hook it into your editor, run two models at once, or figure out why responses started getting weird — and you discover that Ollama alone is not a stack. It's one layer of a stack, and you've been building on bare concrete.
A real local AI setup looks like the diagram above. Eight components working together, each solving a specific problem that no other component solves. Remove any one and you either lose a capability or you're running something that will fail quietly in ways that are hard to debug.
This guide builds the full stack in three tiers. Tier 1 is the beginner setup: Ollama, Open WebUI, and Traefik. Thirty minutes, three containers, works immediately. Tier 2 adds LiteLLM as a proxy layer, Qdrant as a vector store for RAG, and Langfuse for observability.…