跳转至

文章背景与核心概要

稀疏混合专家（MoE）模型通过在每一层使用独立参数化的路由器，将词元（tokens）分派给特定的专家。尽管以往的研究观察到跨层的路由决策表现出一定程度的可预测性，但其底层的结构力学机制长期以来并未得到充分的表征。

本文深入探讨了不同层之间与路由相关的状态如何共享一个通常被层特定坐标系所掩盖的统一几何结构。通过隔离每个路由器的控制子空间，并借助广义正交普罗克鲁斯分析（generalized orthogonal Procrustes analysis）对其进行对齐，作者证明了：单一线性跃迁即可达到 \(0.39\) 至 \(0.71\) 的 \(R^2\) 值，保留了单独拟合的层特定动态中 79%–90% 的预测能力；路由特定信息能够与通用隐藏表示演化实现有效隔离；此外，传输后的规范状态能成功保持局部的路由行为，而学习到的状态演化在 OLMoE 上使负对数似然（\(\Delta\mathrm{NLL}\)）相比简单持续性降低了 15.7%，在 Phi 上跨 10 个路由器跨度降低了 6.2%。

Evidence for Shared Routing Geometry and Dynamics in Sparse Mixture-of-Experts

Authors: Kirill Labzin, Stepan Kulibaba, Artem Dzhalilov, Artem Gorokhov
Submitted: 2 September 2026
Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
Cite as: arXiv:2609.02404 [cs.LG]
DOI: 10.48550/arXiv.2609.02404

Summary

稀疏混合专家（MoE）模型在每个稀疏层利用独立参数化的路由器来为每个词元选择专家。尽管先前的研究表明，跨深度的路由决策通常可以从早期的路由信号中预测出来，这暗示了路由在各层之间并非完全独立，但这种可预测性背后的结构仍然很不清晰。

Sparse mixture-of-experts (MoE) models utilize independently parameterized routers at each layer to assign tokens to specific experts. While previous research has observed that routing decisions display cross-layer predictability, the underlying structural mechanics remained uncharacterized.

本文探讨了跨层的路由相关状态如何共享一个通常被层特定坐标系所掩盖的统一几何结构。通过隔离每个路由器的控制子空间，并使用广义正交普罗克鲁斯分析将其对齐，作者证明了： * 单一线性跃迁可实现 \(0.39\)–\(0.71\) 的 \(R^2\)，保留了单独拟合的层特定动态中 79–90% 的预测能力。 * 路由特定信息可以成功从通用隐藏表示演化中隔离出来（通常残差表示更容易受益）。 * 传输后的规范状态成功保留了局部路由行为，同时在 OLMoE 上，学习到的状态演化使负对数似然（\(\Delta\mathrm{NLL}\)）相比简单持续性降低了 15.7%，在 Phi 模型的 10 路由器视界内降低了 6.2%。

This paper investigates how routing-relevant states across layers share a unified geometric structure that is typically obscured by layer-specific coordinate systems. By isolating the control subspace of each router and aligning them via generalized orthogonal Procrustes analysis, the authors demonstrate that: * A single linear transition achieves an \(R^2\) of \(0.39\)–\(0.71\), preserving 79–90% of the predictive capability found in separately fitted, layer-specific dynamics. * Routing-specific information can be successfully isolated from general hidden representation evolution (which typically benefits residual representations more easily). * Transported canonical states successfully preserve local routing behavior, while learned state evolution reduces negative log-likelihood (\(\Delta\mathrm{NLL}\)) relative to simple persistence by 15.7% on OLMoE and 6.2% over a 10-router horizon on Phi.

Abstract

稀疏混合专家（MoE）模型在每个稀疏层使用独立参数化的路由器，为每个词元选择专家。先前的工作表明，跨深度的路由决策通常可以从早期的路由信号中预测，这表明路由在各层之间并非完全独立。然而，这种可预测性背后的结构仍然不明确。

Sparse mixture-of-experts (MoE) models use an independently parameterized router at each sparse layer to select experts for every token. Prior work has shown that routing decisions across depth can often be predicted from earlier routing signals, suggesting that routing is not fully independent across layers. However, the structure behind this predictability remains unclear.

在这项工作中，我们提供了证据，证明跨层的路由相关状态共享一个共同的几何结构，该结构被层特定的坐标系所模糊。我们隔离了每个路由器的控制子空间，并使用广义正交普罗克鲁斯分析将这些空间对齐到一个共享的规范表示中。对齐后，单个线性跃迁达到了 \(R^2=0.39\)--\(0.71\)，并保留了单独拟合的层特定动态 79--90\% 的预测能力，这表明很大一部分路由状态的演化遵循跨深度的可重用过程。

In this work, we provide evidence that routing-relevant states across layers share a common geometric structure that is obscured by layer-specific coordinate systems. We isolate the control subspace of each router and align these spaces into a shared canonical representation using generalized orthogonal Procrustes analysis. After alignment, a single linear transition reaches \(R^2=0.39\)--\(0.71\) and retains 79--90\% of the predictive power of separately fitted layer-specific dynamics, indicating that much of routing-state evolution follows a reusable process across depth.

然后，我们探讨了这种共享动态是特定于路由的，还是仅仅反映了隐藏表示的平滑演化。匹配秩比较表明，残差表示通常更容易在各层之间进行预测，而路由器控制状态更忠实地保留了模型的专家选择。这就将通用的跨层可预测性与路由特定信息区分开来了。

We then ask whether this shared dynamics is specific to routing or simply reflects the smooth evolution of hidden representations. A matched-rank comparison shows that residual representations are often easier to predict across layers, while router-control states preserve the model's expert choices much more faithfully. This separates generic cross-layer predictability from routing-specific information.

最后，我们测试了当预测的规范状态被用于替代原生路由状态时，它们是否仍然具有意义。传输的状态保留了局部的路由行为，同时在 OLMoE 上，学习到的状态演化使 \(\Delta\mathrm{NLL}\) 相对简单持续性降低了 15.7\%，在 Phi 的 10 路由器视界内降低了 6.2\%。

Finally, we test whether the predicted canonical states remain meaningful when used in place of native routing states. The transported states preserve local routing behavior, while learned state evolution reduces \(\Delta\mathrm{NLL}\) relative to simple persistence by 15.7\% on OLMoE and 6.2\% over a 10-router horizon on Phi.

Full-Text & Resources

External References