Menu

#DraftEr

7 posts

Feed·
7 of 7 posts
Six Serving Engines, One 8 GB GPU: Gemma 4 E2B on an RTX 3050
📰
0Read More

Six Serving Engines, One 8 GB GPU: Gemma 4 E2B on an RTX 3050

DEV Community·xbill·10 days ago
#T30kREUG
#dev#vllm#sglang#drafter#llama

llama.cpp, vLLM, SGLang, mistral.rs, a plain PyTorch engine and LiteRT-LM serve the same 4-bit Gemma 4 E2B on one 8 GB RTX 3050, measured with the same prompts, the same load sweep and the same GSM8K and BFCL questions.…

15s
Six Serving Engines on One EC2 G6: Gemma 4 E2B on an NVIDIA L4
📰
0Read More

Six Serving Engines on One EC2 G6: Gemma 4 E2B on an NVIDIA L4

DEV Community: aws·xbill·12 days ago
#rJLJU2lc
#devtoaws#devto#dev#vllm#drafter

llama.cpp, vLLM, SGLang, mistral.rs, a plain PyTorch engine and LiteRT-LM serve the same 4-bit Gemma 4 E2B on one g6.4xlarge, measured with the same prompts, the same 1-to-64-request sweep and the same GSM8K and BFCL questions.…

15s
Nemotron 3.5 Bolt on one GPU: what the 24 GB covers, the drafter flag and the llama.cpp trap
📰
0Read More

Nemotron 3.5 Bolt on one GPU: what the 24 GB covers, the drafter flag and the llama.cpp trap

DEV Community: opensource·InferenceKey·21 days ago
#zcYSM0ju
#devtoopensource#devto#dev#nvidia#drafter

A 74B vision agent model in 22 GB on one A40: 136 tokens/s with its drafter, 70 without, about 27 GB used. What the 24 GB leaves out.

15s
Size Your Model to Where You Run It: Gemma 4 E2B and E4B on an AWS L4 and Two Small GPUs
📰
0Read More

Size Your Model to Where You Run It: Gemma 4 E2B and E4B on an AWS L4 and Two Small GPUs

DEV Community: aws·xbill·22 days ago
#iwD3vH6N
#devtoaws#devto#dev#drafter#card

A 24 GB NVIDIA L4 serves Gemma 4 E2B and E4B at only 1.2-1.3x an 8 GB desktop RTX 3050 for one user, with the same answers, because a 4-bit small model uses 13-29% of the card. Pick the model for the hardware, and the hardware for the model.

15s
Size Your Model to Where You Run It: Gemma 4 E2B and E4B on an AWS L4 and Two Small GPUs
📰
0Read More

Size Your Model to Where You Run It: Gemma 4 E2B and E4B on an AWS L4 and Two Small GPUs

DEV Community: aws·xbill·22 days ago
#fi4BiesS
#devtoaws#devto#dev#drafter#card

A 24 GB NVIDIA L4 serves Gemma 4 E2B and E4B at only 1.2-1.3x an 8 GB desktop RTX 3050 for one user, with the same answers, because a 4-bit small model uses 13-29% of the card. Pick the model for the hardware, and the hardware for the model.

15s