Put two clients' inference workloads on the same card and one of two things goes wrong: a noisy neighbour destroys the other's latency, or one tenant's memory allocation evicts the other's model. NVIDIA gives you several ways to share a GPU and they are not interchangeable. Two give you throughput but no isolation, one gives you hardware isolation at a fixed grain, and CUDA 13.4 added memory limits that change the arithmetic again. This is the decision table, the Kubernetes configuration for each, and a worked four-H100 layout you can copy.
Multi-tenant GPU platforms fail in a very specific way. Everything looks fine at 20% utilisation, then client A runs a batch job at 14:00, client B's p99 latency triples, and the service level agreement (SLA) credit eats the margin on both accounts.…