Originally published by **Asif Muhammad Iqbal (Founder & CEO at Logic42) * at https://logic42.ai/articles/vllm-disaggregated-prefill-decode-nixlconnector-lmcache .*
Imagine a grocery store routing 64-pallet freight deliveries and ten-item express shoppers onto the exact same checkout belt: every time a truck arrives, forty-eight shoppers mid-checkout freeze for a full second. Running Large Language Model (LLM) prompt ingestion ( prefill ) and token streaming ( decode ) on one Graphics Processing Unit (GPU) node causes that exact stall. Disaggregated Prefill-Decode via NVIDIA Inference Xfer Library ( NIXL ) and LMCache eliminates it, cutting p99 Inter-Token Latency (ITL) by 74% and boosting GPU goodput 2.4x .
Why Colocated Continuous Batching Breaks Under Long-Context RAG
Continuous batching engines like vLLM and SGLang transformed LLM serving by interleaving incoming prompt requests with active token-generation streams inside unified PagedAttention memory.…