[Deep Learning Theory Team Seminar] Understanding the mechanisms of fast hyperparameter transfer (by Nikhil Ghosh, Flatiron Institute)

Speaker: Nikhil Ghosh
Date: 14:00-15:00, 17th/July/2026.
Location: (ハイブリッド開催) 東京大学本郷キャンパス工学部14号館534号室
(セミナー会場へ直接参加可能です)

Title: Understanding the mechanisms of fast hyperparameter transfer
Abstract: The growing scale of deep learning models has rendered standard hyperparameter (HP) optimization prohibitively expensive. A promising solution is the use of scale-aware hyperparameters, which can enable direct transfer of optimal HPs from small-scale grid searches to large models with minimal performance loss. To understand the principles governing such transfer strategy, we develop a conceptual framework for reasoning about HP transfer across scale. In synthetic settings, we present quantitative examples where transfer either offers a provable computational advantage or fails even under μP. To explain the fast transfer observed in practice, we conjecture that decomposing the optimization trajectory reveals two contributions to loss reduction: (1) a width-stable component that determines the optimal HPs and (2) a width-sensitive component that improves with width but weakly perturbs the HP optimum. We present empirical evidence for this hypothesis in large language model pretraining.