Menu

1 / 2

Why is GPU utilization low during PyTorch training?

DEV Community: pytorch·Viswaretas Kotra·9 days ago
#BhniuDHQ
#devtopytorch#devto#dev#time#utilization
Reading 0:00
15s threshold

Low GPU utilization in PyTorch training usually means the GPU is waiting on the CPU: for the next batch, for a sync like .item() , or for Python to launch the next small kernel. On Nodus, as on any platform that bills by the hour, an NVIDIA H100 SXM listed at $2.60/hr on the Nodus pricing page costs the same whether its kernels run 40% or 95% of the time. Measure first with nvidia-smi dmon and torch.profiler , then fix the data loader, the syncs and the batch size, in that order.

What does GPU utilization in nvidia-smi actually measure?

It measures time, not work. NVIDIA's nvidia-smi documentation defines GPU utilization as the "percent of time over the past sample period during which one or more kernels was executing on the GPU", with a sample period between 1/6 of a second and 1 second.

That definition has two consequences. A low number is a clear signal: the GPU had nothing to run for much of each second.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.