AI traffic is volatile, and idle GPU or high-memory instances are expensive. You want replicas to follow demand. The obvious tool is the Horizontal Pod Autoscaler (HPA) , but its default signals, CPU and memory, mislead you for inference servers.
The pipeline
Diagram: FIG 1 · The inference engine exports queue depth, Prometheus stores it, an adapter exposes it to Kubernetes and the HPA adds or removes pods based on it. See the animated version .
Requests reach the inference engine, which exports a queue metric. Prometheus scrapes it, an adapter exposes it to Kubernetes as a custom metric and the HPA adds or removes pods based on it.
Why CPU and memory fail
Diagram: FIG 2 · Inference engines pre-allocate memory and saturate devices even under light load, so CPU and memory utilisation say little about demand. Queue depth and time per output token track what users experience. See the animated version .
Memory.…