2026 10 Muongs

A new paper MuonGS: Pretraining LLMs with Relaxed Weight-Norm Control, joint work with Athanasios and Dawei, is available here. In this work, we use a Gram-Schmidt projection to remove the radial component of each Muon update without pinning the weight norm, show that the resulting weight norm grows at most on the order of the square root of the number of training steps, and obtain lower validation perplexity than MuonH at long training horizons.