Menu

Gemma 4 in Rust on TorchTPU: Hands On With Colab
πŸ“°

Gemma 4 in Rust on TorchTPU: Hands On With Colab

DEV Community: pytorchΒ·xbillΒ·16 days ago
#Jpe0kZH8
#devtopytorch#devto#dev#rust#python
Reading 0:00
15s threshold

This article provides a step by step guide to running Gemma 4 E2B with the model written in Rust and every operation executed by TorchTPU, on a Colab v5e-1 runtime: one TPU v5e chip. A Rust crate reads the checkpoint, unpacks the 4-bit weights, runs the 35 layers and the ring cache, and drives the decode loop. Python hosts the backend and starts the engine. No JAX is involved.

https://github.com/xbill9/torchtpu-dev/tree/main/colab/rust

Rust writes the same tokens as compiled Python at about half the streaming speed and less than half the peak memory. Same chip, same checkpoint, same 19-token prompt, 64 greedy tokens, 33,024-token context:

One Colab v5e-1 chip Streamed tok/s Batched tok/s First call Peak memory

Rust, defer-and-fuse 14.7 to 15.7 15.2 to 16.0 82 to 84 s 7.75 GB

Rust, strict eager 4.8 to 5.0 4.8 23 to 25 s 8.62 GB

Python torch.compile 33.4 to 33.5 45.1 to 45.2 98 to 101 s 16.91 GB

Both Rust ways match the compiled Python model on all 64 tokens and on all three long-prompt answers.

What…

Continue reading β€” create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.