This article provides a step by step guide to running Gemma 4 E2B, Google's quantization-aware trained (QAT) model, in PyTorch on a Google Cloud TPU with TorchTPU. Every step is a Colab notebook that runs on a Colab v5e-1 runtime, one TPU v5e chip with 16 GB of high-bandwidth memory (HBM). The measures are what TorchTPU does with PyTorch code, how far Hugging Face Transformers on TorchTPU trails a JAX engine on the same chip, what closes the gap, and how much context fits.

https://github.com/xbill9/torchtpu-dev/tree/main/colab

One compiled decode step takes PyTorch from 5 to 50 tokens per second, and a ring cache takes it to the full context.…