mini-vllm-rs is a single-node LLM inference engine I built in Rust on top of Candle for macOS. It can run CPU and Metal GPU backends concurrently, routing whole requests between them. My first request routing benchmark favored sending almost everything to GPU: the fastest policies assigned it 11 of 12 requests.

That left a more useful question unanswered: could selective CPU offload improve completion time compared with serving the entire workload on GPU alone? Separating prefill from decode suggested where to look: requests with warm prefixes, arriving behind a growing GPU backlog.

A follow-up experiment tested that idea. Routing 1 of 8 requests to CPU reduced the burst's completion time by 9.8% compared with forcing all 8 onto GPU. With 48 requests, offloading 6 to CPU increased aggregate output throughput by 12.4% .

The phase gap suggested the next experiment

The experiments used a quantized Qwen2.5 0.5B model with F32 activations on an Apple M2.…