This article provides a step by step comparison of six inference serving engines on one Amazon EC2 G6 instance: llama.cpp, vLLM, SGLang, mistral.rs, a plain PyTorch engine and Google's LiteRT-LM, each serving Gemma 4 E2B on Google's QAT 4-bit weights from one NVIDIA L4.
https://github.com/xbill9/gemma4-dev
The question is which engine to put on a G6 when one small model has the whole GPU. Each engine gets the same weights, the same prompts, the same 1-to-64-request load sweep and the same 1,319 GSM8K and 400 BFCL questions, and each is measured with and without Google's E2B drafter wherever the engine supports one. Every engine ran on one instance, one at a time, with the card to itself.
The Results in Brief
Answer quality barely moves between engines. GSM8K lands between 0.8954 and 0.9014 on the five CUDA engines with the drafter on or off, and every paired difference against llama.cpp is within the run-to-run range.β¦