Menu

Six Serving Engines, One 8 GB GPU: Gemma 4 E2B on an RTX 3050
πŸ“°

Six Serving Engines, One 8 GB GPU: Gemma 4 E2B on an RTX 3050

DEV CommunityΒ·xbillΒ·9 days ago
#T30kREUG
#dev#vllm#sglang#drafter#llama
Reading 0:00
15s threshold

This article provides a step by step comparison of six inference serving engines on one consumer GPU: llama.cpp, vLLM, SGLang, mistral.rs, a plain PyTorch engine and Google's LiteRT-LM, each serving Gemma 4 E2B on Google's QAT 4-bit weights from an 8 GB RTX 3050 in a desktop.

https://github.com/xbill9/gemma4-dev

The question is which engine to run when the model lives on a small local card that does nothing else. Each engine gets the same weights, the same prompts, the same 1-to-64-request load sweep and the same 1,319 GSM8K and 400 BFCL questions, and each is measured with and without Google's E2B drafter wherever the engine supports one.

The Results in Brief

Answer quality barely moves between engines. GSM8K lands between 0.8992 and 0.9030 and BFCL between 0.9125 and 0.925 on all six, with the drafter on or off, and on GSM8K every paired difference is within the run-to-run range.…

Continue reading β€” create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.