Menu

Llama vs Qwen vs DeepSeek: How Much VRAM Do You Need to Self-Host?
📰

Llama vs Qwen vs DeepSeek: How Much VRAM Do You Need to Self-Host?

DEV Community: ai·Thea Lauren·12 days ago
#kWfWxuSp
#devtoai#devto#dev#model#vram
Reading 0:00
15s threshold

If you're moving an LLM from a cloud API to your own server, the first thing to calculate is VRAM, not which model scores best.

The two VRAM consumers

1. Model weights : parameters × bytes per parameter.

Precision Bytes per parameter 70B model

FP16/BF16 2 ~140GB

8-bit 1 ~70GB

4-bit 0.5 ~35GB

2. KV cache : grows with context length, batch size and layer count. Llama 3.3 70B uses about 2.6GB for one 8K-token conversation. Ten concurrent users add roughly 26GB on top of the weights, so the total is around 97GB.

How the three families compare

Llama (dense): predictable scaling. The 8B fits on a 24GB card, while the 70B needs 2x A100 80GB or a single H200.

DeepSeek V3/R1 (MoE, 671B): only about 37-40B parameters are active per token, but every parameter must be in memory. At FP8 that is about 746GB, so plan for an 8x H200/B200 node.

Qwen3: the widest range. The 8B model runs at 4-bit on modest hardware.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.