#DraftEr

A 74B vision agent model in 22 GB on one A40: 136 tokens/s with its drafter, 70 without, about 27 GB used. What the 24 GB leaves out.

A 24 GB NVIDIA L4 serves Gemma 4 E2B and E4B at only 1.2-1.3x an 8 GB desktop RTX 3050 for one user, with the same answers, because a 4-bit small model uses 13-29% of the card. Pick the model for the hardware, and the hardware for the model.

A 24 GB NVIDIA L4 serves Gemma 4 E2B and E4B at only 1.2-1.3x an 8 GB desktop RTX 3050 for one user, with the same answers, because a 4-bit small model uses 13-29% of the card. Pick the model for the hardware, and the hardware for the model.

From Dev.to - machinelearning: "Optimizing Multi-Token Prediction with Gemma 4: Insights and Strategies"