TL;DR

On September 21, 2026, Google benchmarked the multi-cluster GKE Inference Gateway at 17,000 compute nodes across the US and Europe, behind one global virtual IP.

The routing signal reads memory, not packets: the EPP exposes KV-cache utilization, and past a 40% threshold traffic spills to the next healthy region automatically.

Measured cost: under 1% overhead at 99.5% of direct local throughput, with near-linear scaling 0.72 → 2.10 req/s across one to three clusters at 99.9% success.

Round-robin treats every request as equal; LLM requests are not. Agentic 100k to 800k+ token contexts exhaust memory before compute — routing must read KV-cache pressure.

Caveats: Google's own benchmark, one MoE model (SGLang) on Google clusters, feature in preview since March 2026 — evidence, not a launch, no SLA.

The reusable model: config cluster outside the request path, KV-cache utilization first-class, LeaderWorkerSet-aware routing to leader pods; claims tagged in the manifest below.

On September 21, 2026, the…