vLLM and SGLang are great inference engines. If you serve chat or batch jobs, use them. But AI agents are a different workload, and I wanted to see how well these engines handle it. So I measured it.

How an agent talks to the model

A coding agent works in turns:

It sends the whole conversation to the model.

The model answers with a tool call (run a command, edit a file).

The agent runs the tool, appends the result, and sends the whole conversation again.

In the traces I used, a median session has about 60 turns and grows to tens of thousands of tokens. Every turn re-sends everything before it.

To answer a turn, the engine needs the KV cache for the full history. If the cache from the last turn is still in GPU memory, only the new tokens need computing.…