Menu

1 / 3

How many GPUs do you need to train an AI model?

DEV Community: machinelearning·Viswaretas Kotra·10 days ago
#cDmQfUmf
Reading 0:00
15s threshold

Most fine-tunes need one GPU, full training of a 70B model needs 8 to 24, and pretraining needs as many as your deadline demands. On Nodus you ask for exactly that shape, from one RTX 4090 listed at $0.29/hr to several 8-GPU machines in one multi-node run (in Beta). The number to ask for is the larger of two floors: enough GPU memory to hold the training state, and enough GPU-hours per day to finish before your deadline.

We built this rule from published memory and compute formulas, and every table below is our own arithmetic, shown so you can redo it for your model. Our posts on VRAM for 70B models and how long training takes go deeper on each floor.

How do you work out how many GPUs a training run needs?

Compute two numbers and take the larger one. The memory floor is the fewest GPUs that can hold the model, gradients and optimizer state.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.