A serving node that pushes 100 Gbps of model traffic does not do it for free. Depending on kernel, MTU and how many small requests you handle, the host burns somewhere between two and six cores on network stack work: interrupts, softirq, socket copies, TLS record processing. On a 64-core machine that is 3% to 9% of the machine, which sounds acceptable until you remember that inference is often memory-bandwidth-bound and those cores are also competing for the same uncore resources.
The obvious answer is to offload the network to a DPU. The obvious answer is also where most teams waste $200,000. Here is the arithmetic that decides it.
Part One: What the host actually does per request
Take a realistic serving configuration: HTTPS, 4 KB average request body, 40 KB response, 300 requests per second per node. Per request the host performs a TLS handshake amortised across connections, parses HTTP headers, copies the body into user space, runs tokenisation, and copies the response back out.…