I put Llama 3.1 8B behind vLLM on a single 24GB card, pointed a load test with 20 concurrent users at it, and watched the stats line:

Running: 4 reqs, Waiting: 16 reqs, GPU KV cache usage: 98.1%

Enter fullscreen mode

Exit fullscreen mode

Four. I had been told vLLM does continuous batching and can run hundreds of sequences at once. The default max_num_seqs is 256. GPU compute utilization was nowhere near maxed. Sixteen users were sitting in a queue anyway.

The answer had been in the startup log the whole time, and it comes down to vLLM KV cache math you can do on a napkin. Once you do it, you stop guessing how many users a GPU can serve.

TL;DR

vLLM's concurrency ceiling is set by KV cache memory , not compute and not max_num_seqs .

KV cache cost per token = 2 × layers × kv_heads × head_dim × bytes .…