Introduction
Earlier in this series I set up the basic platform: vLLM serving the model, LiteLLM as the gateway and an ingress in front. I kept the vLLM flags quite conservative there on purpose. In this part I want to go through what I actually changed afterwards, and why.
My setup: the GPUs are not passed through to the Kubernetes nodes, they are NVIDIA vGPUs on vSphere. Each node VM gets a GRID A100D-40C profile, so half of an A100 with 40 GB of memory. The model used is Qwen3.8-27B in NVFP4 and I want to achieve the best performance for the following requirements:
a context window of 250k tokens
serving many requests in parallel
one 40 GB vGPU
A lot of what follows applies to any GPU, but a few things are really vGPU specific, and those are the ones that cost me the most time.
Info: All the numbers in this post were measured on my hardware with vLLM v0.28.0.…