Menu

1 / 2

Can you use serverless GPUs for long training runs?

DEV Community: machinelearning·Viswaretas Kotra·18 days ago
#22cyhHg4
Reading 0:00
15s threshold

You can, up to a point: Modal caps a single function execution at 24 hours, RunPod Serverless caps a job at 7 days, and on both a preempted or failed run starts over unless your own code saved a checkpoint. Nodus runs long training as a Job or TrainingJob with an optional time limit, saves your state directory automatically, and resumes from the latest checkpoint when capacity is reclaimed. Our rule: serverless is fine for training that finishes well inside one execution and is cheap to repeat; past that, use a platform that owns the checkpoint.

How long can a serverless GPU function run?

Each serverless platform puts a hard ceiling on one execution, and the defaults are much lower than the maximums.

Platform Default limit per execution Maximum What happens at the limit

Modal 5 minutes 24 hours FunctionTimeoutError after any retries

RunPod Serverless 10 minutes 7 days The job fails and the worker stops

Nodus Job or TrainingJob No limit unless you set timeout You can raise or remove it on a running Job…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.