Menu

Hot experts, cold RAM: running a big MoE on a machine with more memory than VRAM
πŸ“°

Hot experts, cold RAM: running a big MoE on a machine with more memory than VRAM

DEV Community: securityΒ·Private Local AIΒ·2 days ago
#MmcbQTZi
#devtosecurity#devto#dev#experts#model
Reading 0:00
15s threshold

Two budgets, and only one of them is expensive

Every local deployment spends against two budgets at once: the size of

the model and the price of the memory holding it. VRAM is the one people

reach for first, and it is the one that costs the most per gigabyte.

System DDR is fast, vast and cheap; GPU memory is fast, small and

expensive. An inference engine that can put weights in the second place

changes the bill.

That is what Strata is for. It is forked from llama.cpp and built for

memory efficiency: it keeps resident only the experts that are hot – the

ones actually in use – and leaves the rest parked. That is how big MoE

models run on small RAM.

Where llama.cpp ends and Strata starts

llama.cpp is the base of everything here: an inference engine in plain

C/C++ that runs GGUF models on hardware you already own – Metal, CUDA,

Vulkan, SYCL or CPU only – with 1.5-8 bit quantisation and hybrid

offload for models too big for the GPU. Hybrid offload moves layers

between CPU and GPU.…

Continue reading β€” create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.