文章背景与核心概要
混合专家（MoE）模型通常通过移除低重要性专家来进行专家剪枝（Expert Pruning），以降低内存和推理服务成本。这一技术传统上依赖于一个假设：路由器的概率能够提供可靠的专家重要性信号。然而，本文指出这一假设在“过度离散路由”（Over-dispersed Routing）机制下会彻底失效——该机制由训练期间激进的负载均衡策略引起，会导致Token被几乎均匀地分发到各个专家，从而使重要性信号崩溃。
为了解决这一问题，研究团队深入分析了过度离散路由带来的困境，发现困惑度（Perplexity）在此机制下无法有效预测下游任务的准确率，且常规的评分指标表现出严重的领域能力权衡。为此，作者提出了极小极大专家分数分配（MESA，Minimax Expert Score Allocation）方法。这是一种领域感知的迭代优化方法，通过提升当前受影响最严重领域的专家分数，最小化最坏情况下的领域性能衰退。实验表明，MESA在25%的专家剪枝率下表现优异，并在多个主流MoE架构（如 gpt-oss-120B、Gemma-4-26B-A4B 和 OLMoE-1B-7B）上展现出良好的泛化能力。
当负载均衡走向极端：过度离散混合专家模型中的专家剪枝
arXiv: 2609.04453 [cs.LG]
提交时间： 2026年9月3日
作者： Berkcan Kapusuzoglu, Connor Pryor, Sangwoo Cho, Supriyo Chakraborty, Shi-Xiong Zhang, Sambit Sahu, Milind Naphade
📌 摘要与核心发现 (Summary)
混合专家（MoE）模型通常采用专家剪枝技术，通过移除低重要性的专家来降低内存和推理服务成本。传统上，这种技术依赖于这样一个假设：路由器的概率能够为专家重要性提供可靠的信号。
本文证明了这一假设在“过度离散路由”下会失效——这是一种由训练期间激进的负载均衡引起的机制，在这种机制下，Token被近乎均匀地分配给各个专家，导致重要性信号崩溃。
核心发现：
- 困惑度崩塌（Perplexity Collapse）： 在过度离散路由（例如
gpt-oss-20B）下，困惑度不再能预测下游任务的准确率。困惑度最低的剪枝配置反而导致了最差的数学推理能力，而困惑度最高的配置却保留了该能力。这与标准路由（例如Mixtral-8x7B-Instruct）形成了鲜明对比，在标准路由中，困惑度和准确率会一同下降。 - 能力权衡（The Capability Trade-Off）： 标准评分指标无法全面占优：激活感知评分（Activation-aware scoring）能够保留数学推理能力，但会严重削弱知识密集型科学任务（在GPQA上暴跌18个百分点），而基于频率的评分则呈现出相反的效果。
- MESA解决方案： 作者引入了极小极大专家服务分配（Minimax Expert Score Allocation, MESA）方法。这是一种领域感知的方法，它通过迭代提升当前受影响最严重的领域所依赖的专家重要性分数。在25%的专家剪枝率下，MESA最小化了最坏情况下的领域退化，在11个基准测试中的7个上超越了激活感知基线，并且能够泛化到多种架构（
gpt-oss-120B、Gemma-4-26B-A4B和OLMoE-1B-7B）。
Mixture-of-Experts (MoE) models often employ expert pruning to reduce memory and serving costs by removing low-importance experts. This technique traditionally relies on the assumption that router probabilities provide a reliable signal of expert importance.
This paper demonstrates that this assumption breaks down under "over-dispersed routing"—a regime caused by aggressive load-balancing during training where tokens are distributed nearly uniformly across experts, collapsing importance signals.
Key Findings:
- Perplexity Collapse: Under over-dispersed routing (e.g.,
gpt-oss-20B), perplexity ceases to predict downstream task accuracy. The lowest-perplexity pruning configuration yields the worst mathematical reasoning, whereas the highest-perplexity configuration preserves it. This contrasts with standard routing (e.g.,Mixtral-8x7B-Instruct), where perplexity and accuracy degrade together.- The Capability Trade-Off: Standard scoring metrics fail to dominate; activation-aware scoring preserves mathematical reasoning but degrades knowledge-intensive science (an 18-point drop on GPQA), while frequency-based scoring yields the opposite effect.
- The MESA Solution: The authors introduce Minimax Expert Score Allocation (MESA), a domain-aware method that iteratively boosts importance scores for experts serving the currently worst-affected domain. At 25% expert pruning, MESA minimizes worst-case domain degradation, outperforming activation-aware baselines on 7 of 11 benchmarks, and generalizes across architectures (
gpt-oss-120B,Gemma-4-26B-A4B, andOLMoE-1B-7B).
📑 论文摘要 (Abstract)
Expert pruning reduces the memory and serving cost of Mixture-of-Experts (MoE) models by removing low-importance experts identified by the router, assuming router probabilities provide a reliable importance signal. We observe that this assumption breaks down under over-dispersed routing, a regime associated with aggressive load-balancing during training, in which tokens are distributed nearly uniformly across experts and importance signals collapse. In this regime, perplexity does not predict downstream task accuracy: on gpt-oss-20B, the lowest-perplexity pruning configuration yields the worst mathematical reasoning, while the highest-perplexity configuration preserves it. This does not occur under standard routing (e.g., Mixtral-8x7B-Instruct), where perplexity and accuracy degrade together. Pruning under over-dispersed routing also exposes a capability trade-off in which no single scoring metric dominates: activation-aware scoring preserves mathematical reasoning but severely degrades knowledge-intensive science (an 18-point gap on GPQA), whereas frequency-based scoring exhibits the reverse. We propose Minimax Expert Score Allocation (MESA), a domain-aware method that iteratively boosts importance scores for experts serving whichever domain is currently worst-affected, minimizing worst-case domain degradation rather than average accuracy. At 25% expert pruning MESA achieves the smallest worst-case degradation across domains, outperforming activation-aware baselines on 7 of 11 benchmarks at a correspondingly reduced memory footprint, and it generalizes to gpt-oss-120B, Gemma-4-26B-A4B, and OLMoE-1B-7B. Our results indicate that over-dispersed routing is a qualitatively distinct pruning regime in which standard assumptions fail, and that recognizing it is a prerequisite for principled expert pruning of load-balanced MoE models.
专家剪枝通过移除由路由器识别出的低重要性专家，来降低混合专家（MoE）模型的内存和推理服务成本，其前提是假设路由器的概率能提供可靠的重要性信号。我们观察到，这一假设在过度离散路由（over-dispersed routing）下会失效。该机制与训练期间激进的负载均衡相关，在其中Token被几乎均匀地分发给各个专家，导致重要性信号崩溃。在此机制下，困惑度无法预测下游任务的准确率：在 gpt-oss-20B 上，困惑度最低的剪枝配置导致了最差的数学推理能力，而困惑度最高的配置却保留了该能力。这在标准路由（例如 Mixtral-8x7B-Instruct）中并不会发生，在标准路由中困惑度和准确率是同步下降的。在过度离散路由下进行剪枝还暴露出一种能力权衡，没有任何单一的评分指标占据主导地位：激活感知评分保留了数学推理能力，但严重降低了知识密集型科学任务的性能（GPQA上有18个百分点的差距），而基于频率的评分则表现相反。我们提出了极小极大专家分数分配（MESA）方法，这是一种领域感知的方法，它迭代地提升服务于当前受影响最严重领域的专家的重要性分数，从而最小化最坏情况下的领域退化，而非仅仅关注平均准确率。在25%的专家剪枝率下，MESA实现了跨领域最小的最坏情况退化，在11个基准测试中的7个上超越了激活感知基线，同时相应地减少了内存占用，并且能够泛化到 gpt-oss-120B、Gemma-4-26B-A4B 和 OLMoE-1B-7B。我们的结果表明，过度离散路由是一个质上截然不同的剪枝机制，传统的标准假设在此失效，而识别这一点是实现有原则的负载均衡MoE模型专家剪枝的前提条件。
📋 附加元数据 (Additional Metadata)
- 学科分类： 机器学习 (
cs.LG)；人工智能 (cs.AI)；计算与语言 (cs.CL) - 引用格式：
arXiv:2609.04453 [cs.LG] - 开源许可： 知识共享署名 4.0 国际许可协议 （见下方许可图标）
