Menu

Tuning vLLM on NVIDIA vGPUs
📰

Tuning vLLM on NVIDIA vGPUs

DEV Community: kubernetes·Florian Lutz·about 2 hours ago
#1V42aM3d
#devtokubernetes#devto#dev#vllm#memory
Reading 0:00
15s threshold

Introduction

Earlier in this series I set up the basic platform: vLLM serving the model, LiteLLM as the gateway and an ingress in front. I kept the vLLM flags quite conservative there on purpose. In this part I want to go through what I actually changed afterwards, and why.

My setup: the GPUs are not passed through to the Kubernetes nodes, they are NVIDIA vGPUs on vSphere. Each node VM gets a GRID A100D-40C profile, so half of an A100 with 40 GB of memory. The model used is Qwen3.8-27B in NVFP4 and I want to achieve the best performance for the following requirements:

a context window of 250k tokens

serving many requests in parallel

one 40 GB vGPU

A lot of what follows applies to any GPU, but a few things are really vGPU specific, and those are the ones that cost me the most time.

Info: All the numbers in this post were measured on my hardware with vLLM v0.28.0.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.