Menu

1 / 5

MIG vs MPS vs time-slicing: how to isolate multiple tenants on one GPU

DEV Community: kubernetes·Azrty·1 day ago
#aBTuWwuP
#devtokubernetes#devto#dev#tenant#nvidia
Reading 0:00
15s threshold

Put two clients' inference workloads on the same card and one of two things goes wrong: a noisy neighbour destroys the other's latency, or one tenant's memory allocation evicts the other's model. NVIDIA gives you several ways to share a GPU and they are not interchangeable. Two give you throughput but no isolation, one gives you hardware isolation at a fixed grain, and CUDA 13.4 added memory limits that change the arithmetic again. This is the decision table, the Kubernetes configuration for each, and a worked four-H100 layout you can copy.

Multi-tenant GPU platforms fail in a very specific way. Everything looks fine at 20% utilisation, then client A runs a batch job at 14:00, client B's p99 latency triples, and the service level agreement (SLA) credit eats the margin on both accounts.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.