2026 09 Uva Talk

M. gave a seminar at the Department of Electrical and Computer Engineering at the University of Virginia, titled “Optimization for Large Language Model Pre-training: New Algorithms, Theory, and Insights”. The talk covered SCALE, a minimalist optimizer built on SGD that matches Adam while using a fraction of its memory; an analysis of why plain SGD struggles in LLM pre-training, and how simple clipping mechanisms close most of the gap; and EMA-Nesterov, a stabilized variant of Nesterov’s lookahead that consistently improves Adam, SOAP and Muon on pre-training benchmarks. The seminar announcement can be found here.