Why We Are Evaluating llm-d
We want an inference gateway to answer a more useful question than “Which backend is next?” We want it to answer: “Which backend can finish this request fastest without repeating expensive work?”
That distinction matters when coding agents repeatedly submit repository context, conversations resend their history, and tool loops carry the same tool definitions across successive requests. A healthy Kubernetes Service can distribute connections while still distributing inference work badly.
Our interest in llm-d is therefore specific: preserve useful KV-cache locality across model replicas without turning a cache-warm replica into a queueing hotspot.
Evidence boundary: this review is a source-backed architecture and deployment assessment, not a completed Effloow GPU benchmark. We did not execute Kubernetes installation or reproduce the Google Cloud and Red Hat performance runs.…