文章背景与核心概要
大语言模型（LLMs）的训练成本极为高昂，因此优化算法的效率至关重要。尽管诸如 Muon 等近期涌现的优化器通过对动量矩阵进行半正交化展现出了卓越的性能，但其成功在很大程度上归因于经验观察，而非坚实的理论基础。
本文引入了一种新颖的物理响应与记忆模型（Physical Response-and-Memory Model），以解释 Muon 优化的底层机制： * “为什么”： 半正交化方向被识别为输出端安全预算下的“最大耗散响应”。 * “记忆”： 动量被建模为在权重矩阵内部累积的内应力，该权重矩阵充当了响应介质。最优记忆长度由该应力的弛豫过程决定。 * Bi-Maxwell 优化器： 基于物理介质在多个时间尺度（快和慢）上发生弛豫的观察，作者提出了 “Bi-Maxwell” 优化器。通过将单时间尺度记忆核替换为双时间尺度方法，他们在显著更少的训练步数内达到了目标损失。
该模型还提供了一个可检验的假设：由于梯度方向在训练初期演化迅速并在后期趋于稳定，因此最优记忆长度应随着训练阶段进行动态缩放。跨越八条独立训练轨迹的实证测量支持了这一结论。
Muon优化器的物理响应与物理记忆模型
A Physical Response-and-Memory Model for Muon Optimization
作者： Yinze Hu, Hongjun Xiang, Xingao Gong, Hongyu Yu
日期： 2026年8月24日
arXiv： 2608.22994
学科： 机器学习 (cs.LG)；人工智能 (cs.AI)；统计力学 (cond-mat.stat-mech)
总结
Summary
Training large language models (LLMs) is computationally expensive, making the efficiency of optimization algorithms critical. While recent optimizers like Muon have demonstrated superior performance by semi-orthogonalizing the momentum matrix, their success has largely been attributed to empirical observation rather than theoretical grounding.
本文引入了一种新颖的物理响应与记忆模型，用以解释 Muon 优化的机制： * “为什么”： 半正交化方向被确立为输出端安全预算下的“最大耗散响应”。 * “记忆”： 动量被建模为在权重矩阵内部积累的内应力，该权重矩阵起到了响应介质的作用。最优记忆长度由该应力的弛豫决定。 * Bi-Maxwell 优化器： 基于物理介质跨多个时间尺度（快与慢）弛豫的观察，作者提出了 “Bi-Maxwell” 优化器。通过将单一时间尺度的记忆核替换为双时间尺度方法，他们在显著更少的训练步数内达到了目标损失。
This paper introduces a novel Physical Response-and-Memory Model to explain the mechanics of Muon optimization: * The "Why": The semi-orthogonalized direction is identified as the "maximally dissipative response" under an output-side safety budget. * The "Memory": Momentum is modeled as internal stress accumulated within the weight matrix, which acts as a responsive medium. The optimal memory length is determined by the relaxation of this stress. * The Bi-Maxwell Optimizer: Based on the observation that physical media relax across multiple timescales (fast and slow), the authors propose the "Bi-Maxwell" optimizer. By replacing a single-timescale memory kernel with a dual-timescale approach, they achieve target loss in significantly fewer training steps.
该模型还提供了一个可检验的假设：由于梯度方向在训练初期演化迅速，后期趋于稳定，因此最优记忆长度应该随训练阶段进行缩放。对八条独立训练轨迹的实证测量支持了这一结论。
The model also provides a testable hypothesis: because gradient directions evolve rapidly early in training and stabilize later, the optimal memory length should scale with the training stage. Empirical measurements across eight independent training trajectories support this conclusion.
核心贡献
Key Contributions
- 理论框架： 通过将权重视为物理介质，弥合了神经网络优化与统计力学之间的鸿沟。
- 机理洞察： 从耗散响应的视角解释了半正交化的有效性。
- 算法创新： 引入了 Bi-Maxwell 优化器，通过结合多时间尺度记忆核提升了训练效率。
- 实证验证： 证明了其在公开大语言模型基准测试中的性能提升，并为所提出的记忆长度缩放提供了实验证据。
- Theoretical Framework: Bridges the gap between neural network optimization and statistical mechanics by treating weights as a physical medium.
- Mechanistic Insight: Explains the efficacy of semi-orthogonalization through the lens of dissipative response.
- Algorithmic Innovation: Introduces the Bi-Maxwell optimizer, which improves training efficiency by incorporating multi-timescale memory kernels.
- Empirical Validation: Demonstrates performance gains on public large-language-model benchmarks and provides experimental evidence for the proposed memory-length scaling.
获取与资源
Access & Resources
