Menu

How do you give each customer an isolated cluster and let their teams share idle GPUs? Tested on bare-metal H100s

Reddit r/kubernetes·u/dariotranchitella·3 days ago
#s7KUlt3n
#idle#customer#gpus#anish#cluster
Reading 0:00
15s threshold

Disclosure: I'm the maintainer of Kamaji, and I lent Anish the H100s and the bare-metal infrastructure. He wrote and ran everything himself, so treat me as biased and verify the numbers.

Anish Bista wrote up a lab combining Kamaji (a control plane per customer, for isolation) with Run:ai/KAI projects (per team, for GPU sharing).

We decided to work together on this topic because it's a question that comes up a lot when prospects aim to sell GPUs to more than one customer: separate clusters solve " can they see my workloads?" but not " my fraud team's GPUs sit idle all night while my NLP team queues ".

Anish ran it on two bare-metal servers with one H100 80GB each (the journey to find them could be worth another interesting blog post) and tested borrowing and reclaiming between teams, fractional GPU memory limits, scheduling rules, and whether a busy tenant can use another tenant's idle card (it can't).

Some findings are less flattering than the title, which is why I like it: the idle-GPU timeout only…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.