Menu

1 / 2

Which GPU platforms checkpoint and resume training for you?

DEV Community: machinelearning·Viswaretas Kotra·20 days ago
#55yTZPiF
Reading 0:00
15s threshold

We checked five platforms against their own docs. Nodus, AWS SageMaker managed spot training and Modal all restart a preempted training run and hand it back the files it saved, but only Nodus also tells your program when to save, including an urgent request when capacity is about to be reclaimed. RunPod Pods and Vast.ai interruptible instances keep your disk and leave the restart to you.

One thing is true everywhere: no platform resumes your training loop for you. Your script still has to look for its last checkpoint when it starts and load it.

What differs is everything around that load, and that is the part that decides whether a run finishes unattended. This post compares the platforms. For the code itself, see how to make spot GPU training survive preemption .

Who saves, who restarts and who resumes?

Recovery after a lost GPU has five steps. Something has to trigger a save, copy it off the machine, start a new machine, put the files back and continue the loop.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.