Training LLMs at massive scale brings unique infrastructure challenges, especially as jobs span thousands of GPUs and run for extended periods. The longer these jobs run, the greater the likelihood of encountering unscheduled interruptions or resource fluctuations. Even infrequent device unavailability can have outsized effects on tightly interconnected clusters, resulting in slowdowns for a given training run.

For large-scale training jobs, elastically adapting the job to the number of available GPUs is a powerful method to improve Goodput . In the context of AI training, Goodput represents the critical measure of useful, convergence-driving work completed, rather than just raw hardware throughput.

Effective methods for elastic scaling today include dropping a data replica, utilizing fast checkpoint-restarts, or swapping to hot spares. These methods enable LLM jobs to adapt to GPU availability changes while maintaining balance across the system.…