6000 tokens from a vLLM server on a private GKE GPU node reached my Mac in 76.7 seconds through a plain kubectl port-forward , with nothing in between but Google's DNS-based endpoint for the cluster. The model is for me, so I didn't want an Ingress or a bastion VM for it. I tried port-forward through both the DNS-based endpoint and the Connect gateway.
The cluster is a Standard GKE cluster in us-east4, on 1.36.4-gke.2046003 when I ran these tests. Its IP endpoints are disabled and its DNS-based endpoint allows external traffic, so the only way in from outside is the DNS name, gated by IAM.…