I'm in the UK and recently I've been working on a client project where UK or EU data residency is desirable. We need gemini to do some things for us, but when using either london, EU, or global endpoints, vertex gemini is constantly giving 429 out of capacity errors.

Looking at the metrics, I can see that our request rate is very small, roughly 0.5 RPS, max of about 2. But the 429 error rate is up to 80%.

At first I thought it was a glitch. Maybe related to London's outage the other day. But this has been happening for more than a week now and continues today. It changes throughout the day, as you'd expect, but still very very high error rates.

On a previous project we never had these issues. The only difference was that this client was US-based and we were using the us endpoint (not global). So this morning I switched to that and boom, ~0.5-1% error rate.

What I don't get is why the internet isn't complaining about this?…