Menu

Tensor vs. Pipeline vs. Data Parallelism: How LLM Inference Scales Across GPUs
πŸ“°

Tensor vs. Pipeline vs. Data Parallelism: How LLM Inference Scales Across GPUs

DEV CommunityΒ·Ankit SΒ·10 days ago
#rUJRDuwF
#dev#model#parallelism#stage#every
Reading 0:00
15s threshold

Sooner or later, every LLM deployment hits one of two walls. Either the model does not fit on one GPU β€” a 70B model in FP16 needs roughly 140 GB of weights before you allocate a single byte of KV cache β€” or it fits, but one GPU cannot serve your traffic at the latency you promised. Both walls are broken the same way: by spreading the work across GPUs. But "spread it across GPUs" hides three very different strategies, and choosing the wrong one buys you expensive hardware doing slow work.

The three strategies answer three different questions:

Tensor parallelism: how do I make one request faster (or possible at all)?

Pipeline parallelism: how do I make a model that spans machines workable?

Data parallelism: how do I serve more requests at once ?

This post walks through how each one actually splits the work, what it costs in communication, and how they combine in real deployments β€” in vLLM terms, the flags behind --tensor-parallel-size , --pipeline-parallel-size , and --data-parallel-size .

Tensor…

Continue reading β€” create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.