Notes from a 2 AM On-Call: Why Our vLLM Semantic Router Dropped 40% of Requests Under Load

The Failure Symptom

2024-03-15 02:17:43 WARNING: Route decision timeout after 5000ms 2024-03-15 02:17:48 ERROR: 127 requests dropped (no route assigned) 2024-03-15 02:18:12 INFO: QPS: 240 → 140 (41% degradation)

Enter fullscreen mode

Exit fullscreen mode

Our FastAPI gateway wrapping vllm-project/semantic-router started silently dropping requests at 2 AM during a traffic spike. No 5xx errors, no OOM kills, just requests vanishing into the void. Prometheus showed route decision latency climbing from 80ms to 4.8 seconds, then requests just disappeared from logs entirely.

The Red Herring

The semantic-router README shows clean synchronous examples:

from semantic_router import Router

router = Router () route = router ( " Which model is best for code generation?…