"Will it fit on my card?" comes up constantly in local-LLM threads, and the usual answer is a rule of thumb like "about half a gigabyte per billion parameters at 4-bit." That rule is close enough for small dense models. For mixture-of-experts models, models with sliding-window attention, and long contexts it can be badly off.
So instead of a rule of thumb, I put together real numbers. For 30 popular open-weight models I recorded the actual GGUF download sizes from Hugging Face and the KV-cache size per token, worked out from each model's config.json . This post uses the 26 models that have both a 4-bit file and a sourced KV-cache figure.
All the data is open (CC BY 4.0) at huggingface.co/datasets/vramneeded/llm-vram-requirements . It's the same data behind the comparison table on vramneeded.com .
The method: weights + KV cache + overhead
The memory a local model uses during inference comes from three parts:
Weights. I use the actual GGUF file size, not params Γ bits.β¦