Fix a CUDA out of memory error by measuring what holds the memory, then cutting the biggest consumer first: leaked tensors, fragmentation, activations, and finally the weights, gradients and optimizer state. When the arithmetic says the job still cannot fit, Nodus runs the same script on a GPU that can. Set a memory floor such as minMemory: 80Gi instead of a GPU name, and Nodus picks the accelerator that is cheapest to finish the job.
This post gives one order of fixes, the gigabytes each step buys when fine-tuning Llama 3.1 8B on 4,096-token sequences, and code you can paste. The figures are our own arithmetic from the model's config and published formulas. They are estimates, so check them against your own peak numbers.
What is the OOM error actually telling you?
The error message already says which kind of problem you have.…