Menu

Gemma 4 E2B in JAX on One TPU v5e: bf16 or 4-Bit, Python or Rust, and How It Stands Against vLLM
πŸ“°

Gemma 4 E2B in JAX on One TPU v5e: bf16 or 4-Bit, Python or Rust, and How It Stands Against vLLM

DEV Community: rustΒ·xbillΒ·12 days ago
#SrvR006u
#devtorust#devto#dev#engine#rust
Reading 0:00
15s threshold

This article provides a step by step guide to serving Gemma 4 E2B on a single Cloud TPU v5e chip with JAX. A pure-JAX engine serves the model from Python, a second engine builds the same graph in Rust and hands it to the libtpu compiler that JAX itself uses, and vLLM serves the same checkpoint as the reference. Every number below was measured on one flex-start v5e-1 VM on 2026-11-03, and every run is committed with its logs.

https://github.com/xbill9/jax-dev

Why One v5e Chip?

A v5e-1 is the smallest Cloud TPU you can rent: one chip, 16.909 GB of HBM, in a ct5lp-hightpu-1t VM. Gemma 4 E2B in bf16 puts 9.257 GB of weights on it, which leaves room for the KV cache with no quantization at all.…

Continue reading β€” create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.