Frontier model pre-training has converged on mixture of experts (MoE), which is fundamentally changing what limits large-scale AI training. As compute per token falls, communication increasingly determines how efficiently models scale across thousands of GPUs. NVIDIA GB300 NVL72 set a world record for pre-training DeepSeek-V3 671B at 1,648 TFLOPs per GPU, showing how advances across the entire AI platform—from silicon to networking to software—continue to push training performance forward. Every gain in pre-training efficiency means researchers can train larger models, run more experiments, and reach frontier capability faster on the same NVIDIA infrastructure.
The rapid industry shift toward these MoE architectures is driven by their massive computational efficiency. Unlike dense models, where every token activates every parameter and compute per token rises with the total parameter count, MoE models activate a subset of parameters for each token.…