Menu

Gemma 4 in Rust on TorchTPU: A Native TPU Rig With PyO3 and MCP
πŸ“°

Gemma 4 in Rust on TorchTPU: A Native TPU Rig With PyO3 and MCP

DEV Community: pytorchΒ·xbillΒ·13 days ago
#WSyr91xa
#devtopytorch#devto#dev#define#rust
Reading 0:00
15s threshold

This article provides a step by step guide to serving Gemma 4 E2B from a Cloud TPU v5e-1 VM with the model written in Rust and every operation run by TorchTPU. A Claude Code MCP server provisions the TPU, the VM builds the Rust crate on boot and serves an OpenAI-compatible endpoint, and a local shim gives it a chat page and a second MCP server for demos. Python hosts the process and the HTTP server; the 35 layers, the ring cache and the decode loop are Rust.

https://github.com/xbill9/torchtpu-dev/tree/main/tpu-pytorchrust-v5e1-2b-q4w4a16emb4

The Rust model reaches TorchTPU through a 154-line C++ shim over ATen, so it builds against whichever PyTorch is installed: here Python 3.14, PyTorch 2.14.1 and a nightly Rust.…

Continue reading β€” create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.