GPU clusters treat a GPU as healthy or broken. Real GPUs also end up slow : a lowered
power limit, a clock lock, heat. That middle state is expensive in a specific way: in
synchronous distributed training every step waits for the slowest GPU, so one slow GPU
slows every GPU in the job, and you pay for all of them.
The usual answer is to remove the slow GPU. But "slow" depends on the work. So I built
GPUFit , an open-source tool that grades every GPU per workload (training, fine-tuning,
inference) and uses that grade for scheduling in Kubernetes: a degraded GPU is kept out of
synchronous training and keeps serving inference.
Everything below was measured on real NVIDIA GPUs on AWS.…