Low GPU utilization in PyTorch training usually means the GPU is waiting on the CPU: for the next batch, for a sync like .item() , or for Python to launch the next small kernel. On Nodus, as on any platform that bills by the hour, an NVIDIA H100 SXM listed at $2.60/hr on the Nodus pricing page costs the same whether its kernels run 40% or 95% of the time. Measure first with nvidia-smi dmon and torch.profiler , then fix the data loader, the syncs and the batch size, in that order.
What does GPU utilization in nvidia-smi actually measure?
It measures time, not work. NVIDIA's nvidia-smi documentation defines GPU utilization as the "percent of time over the past sample period during which one or more kernels was executing on the GPU", with a sample period between 1/6 of a second and 1 second.
That definition has two consequences. A low number is a clear signal: the GPU had nothing to run for much of each second.…