To train on AWS spot instances without losing work, save checkpoints to storage that outlives the instance, reload them on start, and restart the job after each interruption. Amazon SageMaker managed spot training handles the restart and S3 sync, while plain Amazon EC2 Spot leaves all of it to you. A Nodus job checkpoints and restores itself, including when it runs in your connected AWS account.
The part most guides skip is the two-minute warning. It is only useful if your checkpoint can leave the machine inside it, and for a full fine-tune of a large model it usually can't. Below we show the sizes, the setup for each path, and the rule we use to pick one.
What actually happens when AWS reclaims a spot instance?
You get about two minutes, and then the instance is terminated unless you asked for something else. AWS issues an interruption notice two minutes before it stops or terminates a Spot Instance , as an Amazon EventBridge event and as the spot/instance-action item in instance metadata.…