This article provides a step by step comparison of six inference serving engines on one consumer GPU: llama.cpp, vLLM, SGLang, mistral.rs, a plain PyTorch engine and Google's LiteRT-LM, each serving Gemma 4 E2B on Google's QAT 4-bit weights from an 8 GB RTX 3050 in a desktop.
https://github.com/xbill9/gemma4-dev
The question is which engine to run when the model lives on a small local card that does nothing else. Each engine gets the same weights, the same prompts, the same 1-to-64-request load sweep and the same 1,319 GSM8K and 400 BFCL questions, and each is measured with and without Google's E2B drafter wherever the engine supports one.
The Results in Brief
Answer quality barely moves between engines. GSM8K lands between 0.8992 and 0.9030 and BFCL between 0.9125 and 0.925 on all six, with the drafter on or off, and on GSM8K every paired difference is within the run-to-run range.β¦