At low traffic, a model server is mostly idle. The obvious optimization — batching — is also the one people skip because it feels like a high-traffic problem.

But even at a few requests per second, dynamic batching smoothed out our p99 dramatically. The trick was making it not add latency: a 10ms window, max batch size 8.

We ended up writing a tiny batching layer in front of the model, then moved it into the service once the pattern was clear.

I'm learning Rust to see what a zero-GC serving layer looks like. If you've built something similar, I'd love to compare notes.