When GPU capacity runs out before a training job starts, the job fails with a capacity error or waits in a queue; when it runs out mid-run, you lose everything since your last checkpoint. Nodus handles both cases: a job with no matching capacity waits in Queued for up to 24 hours by default, and a reclaimed job resumes from its latest checkpoint on new capacity. How much a reclaim costs you comes down to how often you save.
This post prices one reclaim on a 10-hour, 8 x H100 run, lists the warning each major cloud gives before it takes a machine back, and gives a rule for choosing a checkpoint interval.
What happens if there is no GPU free when you launch?
On a raw cloud account, the launch call fails and retrying is your job. Amazon EC2 returns InsufficientInstanceCapacity when it lacks the On-Demand capacity you asked for, and its advice is to wait a few minutes or ask for fewer instances per request.…