Menu

Six Serving Engines on One EC2 G6: Gemma 4 E2B on an NVIDIA L4
πŸ“°

Six Serving Engines on One EC2 G6: Gemma 4 E2B on an NVIDIA L4

DEV Community: awsΒ·xbillΒ·11 days ago
#rJLJU2lc
#devtoaws#devto#dev#vllm#drafter
Reading 0:00
15s threshold

This article provides a step by step comparison of six inference serving engines on one Amazon EC2 G6 instance: llama.cpp, vLLM, SGLang, mistral.rs, a plain PyTorch engine and Google's LiteRT-LM, each serving Gemma 4 E2B on Google's QAT 4-bit weights from one NVIDIA L4.

https://github.com/xbill9/gemma4-dev

The question is which engine to put on a G6 when one small model has the whole GPU. Each engine gets the same weights, the same prompts, the same 1-to-64-request load sweep and the same 1,319 GSM8K and 400 BFCL questions, and each is measured with and without Google's E2B drafter wherever the engine supports one. Every engine ran on one instance, one at a time, with the card to itself.

The Results in Brief

Answer quality barely moves between engines. GSM8K lands between 0.8954 and 0.9014 on the five CUDA engines with the drafter on or off, and every paired difference against llama.cpp is within the run-to-run range.…

Continue reading β€” create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.