Menu

1 / 2

How much does it cost to train an LLM from scratch?

DEV Community: machinelearning·Viswaretas Kotra·16 days ago
#gFzXSmQG
Reading 0:00
15s threshold

Training an LLM from scratch costs from a few hundred dollars for a small model to tens of millions for a frontier model, and almost all of it is GPU-hours times the hourly rate. At the $2.60/hr Nodus lists for an NVIDIA H100 on its pricing page , a compute-optimal 1B model costs about $292 and a 7B model about $14,300 in compute. The rest of this post shows the formula, checks it against what Meta and DeepSeek actually published, and lists the costs the headline numbers leave out.

How do you estimate what an LLM training run will cost?

You need three numbers: model size, training tokens and how much of the GPU's peak speed you will really get. Training a transformer takes about 6 floating-point operations per parameter per token, the rule from Kaplan et al.'s scaling-laws paper . So a run needs roughly 6 × parameters × tokens operations in total.

An NVIDIA H100 SXM peaks at about 989 teraFLOPS of dense BF16 math.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.