Menu

How much VRAM do popular LLMs really need at 4-bit and 8K context? Data from 26 models
πŸ“°

How much VRAM do popular LLMs really need at 4-bit and 8K context? Data from 26 models

DEV Community: machinelearningΒ·Gary NagraΒ·17 days ago
#guG2pOjn
#devtomachinelearning#education#devto#dev#cache
Reading 0:00
15s threshold

"Will it fit on my card?" comes up constantly in local-LLM threads, and the usual answer is a rule of thumb like "about half a gigabyte per billion parameters at 4-bit." That rule is close enough for small dense models. For mixture-of-experts models, models with sliding-window attention, and long contexts it can be badly off.

So instead of a rule of thumb, I put together real numbers. For 30 popular open-weight models I recorded the actual GGUF download sizes from Hugging Face and the KV-cache size per token, worked out from each model's config.json . This post uses the 26 models that have both a 4-bit file and a sourced KV-cache figure.

All the data is open (CC BY 4.0) at huggingface.co/datasets/vramneeded/llm-vram-requirements . It's the same data behind the comparison table on vramneeded.com .

The method: weights + KV cache + overhead

The memory a local model uses during inference comes from three parts:

Weights. I use the actual GGUF file size, not params Γ— bits.…

Continue reading β€” create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.