Menu

Port-forwarding to a model on a private GKE GPU cluster: DNS endpoint or Connect gateway
📰

Port-forwarding to a model on a private GKE GPU cluster: DNS endpoint or Connect gateway

DEV Community: kubernetes·Edgar·4 days ago
#5JQjCEfo
#devtokubernetes#devto#dev#forward#kubectl
Reading 0:00
15s threshold

6000 tokens from a vLLM server on a private GKE GPU node reached my Mac in 76.7 seconds through a plain kubectl port-forward , with nothing in between but Google's DNS-based endpoint for the cluster. The model is for me, so I didn't want an Ingress or a bastion VM for it. I tried port-forward through both the DNS-based endpoint and the Connect gateway.

The cluster is a Standard GKE cluster in us-east4, on 1.36.4-gke.2046003 when I ran these tests. Its IP endpoints are disabled and its DNS-based endpoint allows external traffic, so the only way in from outside is the DNS name, gated by IAM.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.