Why You Need an AI Gateway
The first LLM feature ships fast: one API key, one provider, one SDK call. Six months later you have:
4 services calling 3 different providers
API keys scattered across repos
No idea which team is burning the budget
An outage at one provider that took down your product
An AI gateway is a single entry point between your applications and LLM providers. It's the same idea as an API gateway, but it handles problems specific to LLMs: token-based billing, slow streaming responses, flaky providers, and non-deterministic outputs.
In this post we'll design one from scratch.
Requirements
Functional
One unified API (OpenAI-compatible) for all providers
Route requests across providers and models
Automatic retries and failover
Per-team API keys, quotas, and rate limits
Response caching
Streaming (SSE) support
Non-functional
Added latency overhead under 10 ms (p99)
No single point of failure
Full visibility into cost, latency, and errors per request
Provider keys neverβ¦