Most "will it fit" questions about self-hosted models get answered with a rule of thumb: two bytes per parameter, add some headroom, done. That rule tells you whether the weights load. It does not tell you whether the server survives fifty concurrent users with long prompts, which is the question that actually decides what card you rent.

This post walks through the arithmetic I use before picking hardware for inference and fine-tuning, with a small Python calculator, a working vLLM launch config and the mistakes that cost the most time.

The four things that live in VRAM

During inference, GPU memory is split between:

Model weights. Fixed size, determined by parameter count and precision.

KV cache. Grows with context length and with the number of sequences served in parallel. This is the part people forget.

Activations and temporary buffers. Relatively small for inference, but spikes during prefill of long prompts.

Runtime overhead. CUDA context, cuBLAS workspaces, NCCL buffers if you use several cards.…