You SSH into your pod. You run your training script. The loss starts printing. Everything looks normal.
Except it is running at 24 seconds per iteration instead of 0.4.
That is how I found out torch.cuda.is_available() had returned False on a Vast.ai pod a failure that hits RunPod users equally. No exception. No warning. My script resolved to CPU and kept running. I was paying full GPU rates and training on CPU the entire time.
Why this happens without an error
torch.cuda.is_available() returns False when PyTorch cannot find a usable CUDA installation. The reasons vary:
PyTorch was installed without CUDA support a CPU-only build (torch+cpu) instead of a CUDA build (torch+cu121, torch+cu124 etc)
The host NVIDIA driver does not meet the compatibility requirements for the CUDA runtime PyTorch was compiled against
CUDA_VISIBLE_DEVICES is set to an empty string or -1, hiding all GPUs from PyTorch the GPU may still appear in nvidia-smi but PyTorch cannot see it
The NVIDIA driver failed to load correctly on…