Two budgets, and only one of them is expensive
Every local deployment spends against two budgets at once: the size of
the model and the price of the memory holding it. VRAM is the one people
reach for first, and it is the one that costs the most per gigabyte.
System DDR is fast, vast and cheap; GPU memory is fast, small and
expensive. An inference engine that can put weights in the second place
changes the bill.
That is what Strata is for. It is forked from llama.cpp and built for
memory efficiency: it keeps resident only the experts that are hot β the
ones actually in use β and leaves the rest parked. That is how big MoE
models run on small RAM.
Where llama.cpp ends and Strata starts
llama.cpp is the base of everything here: an inference engine in plain
C/C++ that runs GGUF models on hardware you already own β Metal, CUDA,
Vulkan, SYCL or CPU only β with 1.5-8 bit quantisation and hybrid
offload for models too big for the GPU. Hybrid offload moves layers
between CPU and GPU.β¦