This article provides a step by step guide to running Gemma 4 E2B with the model written in Rust and every operation executed by TorchTPU, on a Colab v5e-1 runtime: one TPU v5e chip. A Rust crate reads the checkpoint, unpacks the 4-bit weights, runs the 35 layers and the ring cache, and drives the decode loop. Python hosts the backend and starts the engine. No JAX is involved.
https://github.com/xbill9/torchtpu-dev/tree/main/colab/rust
Rust writes the same tokens as compiled Python at about half the streaming speed and less than half the peak memory. Same chip, same checkpoint, same 19-token prompt, 64 greedy tokens, 33,024-token context:
One Colab v5e-1 chip Streamed tok/s Batched tok/s First call Peak memory
Rust, defer-and-fuse 14.7 to 15.7 15.2 to 16.0 82 to 84 s 7.75 GB
Rust, strict eager 4.8 to 5.0 4.8 23 to 25 s 8.62 GB
Python torch.compile 33.4 to 33.5 45.1 to 45.2 98 to 101 s 16.91 GB
Both Rust ways match the compiled Python model on all 64 tokens and on all three long-prompt answers.
Whatβ¦