Menu

A slow GPU isn't a broken GPU: grading GPUs per workload in Kubernetes (measured on real A10Gs)
📰

A slow GPU isn't a broken GPU: grading GPUs per workload in Kubernetes (measured on real A10Gs)

DEV Community: kubernetes·praveen9161·about 1 hour ago
#KGuVdwep
#devtokubernetes#devto#dev#gpus#training
Reading 0:00
15s threshold

GPU clusters treat a GPU as healthy or broken. Real GPUs also end up slow : a lowered

power limit, a clock lock, heat. That middle state is expensive in a specific way: in

synchronous distributed training every step waits for the slowest GPU, so one slow GPU

slows every GPU in the job, and you pay for all of them.

The usual answer is to remove the slow GPU. But "slow" depends on the work. So I built

GPUFit , an open-source tool that grades every GPU per workload (training, fine-tuning,

inference) and uses that grade for scheduling in Kubernetes: a degraded GPU is kept out of

synchronous training and keeps serving inference.

Everything below was measured on real NVIDIA GPUs on AWS.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.