Sooner or later, every LLM deployment hits one of two walls. Either the model does not fit on one GPU β a 70B model in FP16 needs roughly 140 GB of weights before you allocate a single byte of KV cache β or it fits, but one GPU cannot serve your traffic at the latency you promised. Both walls are broken the same way: by spreading the work across GPUs. But "spread it across GPUs" hides three very different strategies, and choosing the wrong one buys you expensive hardware doing slow work.
The three strategies answer three different questions:
Tensor parallelism: how do I make one request faster (or possible at all)?
Pipeline parallelism: how do I make a model that spans machines workable?
Data parallelism: how do I serve more requests at once ?
This post walks through how each one actually splits the work, what it costs in communication, and how they combine in real deployments β in vLLM terms, the flags behind --tensor-parallel-size , --pipeline-parallel-size , and --data-parallel-size .
Tensorβ¦