Menu

1 / 2

DPU Offload vs Host Cores: When the 100G NIC Stops Being Free

DEV Community: cloud·gxqwb·13 days ago
#S7pBlpQm
#devtocloud#devto#dev#node#network
Reading 0:00
15s threshold

A serving node that pushes 100 Gbps of model traffic does not do it for free. Depending on kernel, MTU and how many small requests you handle, the host burns somewhere between two and six cores on network stack work: interrupts, softirq, socket copies, TLS record processing. On a 64-core machine that is 3% to 9% of the machine, which sounds acceptable until you remember that inference is often memory-bandwidth-bound and those cores are also competing for the same uncore resources.

The obvious answer is to offload the network to a DPU. The obvious answer is also where most teams waste $200,000. Here is the arithmetic that decides it.

Part One: What the host actually does per request

Take a realistic serving configuration: HTTPS, 4 KB average request body, 40 KB response, 300 requests per second per node. Per request the host performs a TLS handshake amortised across connections, parses HTTP headers, copies the body into user space, runs tokenisation, and copies the response back out.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.