MuonGS: Squeezing 15% More Speed Out of LLM Pretraining via Relaxed Weight-Norm Control

In the high-stakes world of Large Language Model (LLM) pretraining, efficiency is everything. When you are training a model on trillions of tokens, a 5% improvement in optimizer efficiency can save millions of dollars in compute costs. Today, a new contender has emerged in the "Muon" family of optimizers: MuonGS .

Published on October 27, 2026, by researchers Athanasios Glentis, Dawei Li, and Mingyi Hong, MuonGS (Gram-Schmidt) introduces a sophisticated way to manage weight-norm growth. By relaxing the strict constraints found in previous versions of the optimizer, MuonGS delivers an estimated 15% token-equivalent speedup on dense models and shows strong results on Mixture-of-Experts (MoE) architectures.

The Problem: The Effective Learning Rate Decay

To understand why MuonGS is such a significant step forward, we first need to look at the "effective learning rate" problem.…