跳转至

文章背景与核心概要

混合专家（MoE）架构能够高效扩展大语言模型（LLMs），但标准的固定 top-\(k\) 路由机制会为每个 token 激活完全相同数量的专家槽位，从而导致不必要的计算冗余。现有的专家跳过方法往往依赖于路由器置信度、成本高昂的校准数据集或额外的模型训练，这阻碍了它们准确评估专家真实贡献的能力。

为了解决这些局限性，本文推出了 ACE，这是一个专为基于 MoE 的大语言模型设计的免训练、免校准且保持模型检查点不变的框架，用于实现 token 自适应的专家跳过。

ACE: Adaptive Calibration-Free Expert Skipping for MoE-based LLMs

  • arXiv ID: 2609.05228 [cs.AI]
  • Submitted: September 4, 2026
  • Authors: Zukang Xu, Zhixiong Zhao, Xing Hu, Jiangyong Yu, Houji Wen, Jun Li, Zhe Jiang, Dawei Yang
  • Links: View PDF | HTML Version
  • arXiv ID: 2609.05228 [cs.AI]
  • Submitted: September 4, 2026
  • Authors: Zukang Xu, Zhixiong Zhao, Xing Hu, Jiangyong Yu, Houji Wen, Jun Li, Zhe Jiang, Dawei Yang
  • Links: View PDF | HTML Version

📌 Summary

Mixture-of-Experts (MoE) architectures scale Large Language Models (LLMs) efficiently, but standard fixed top-\(k\) routing activates the exact same number of expert slots for every token, leading to unnecessary computational redundancy. Existing expert-skipping approaches often rely on router confidence, costly calibration datasets, or additional model training, which hinders their ability to accurately estimate an expert's true contribution.

To resolve these limitations, the paper introduces ACE, a training-free, calibration-free, and checkpoint-preserving framework designed for token-adaptive expert skipping in MoE-based LLMs.

📌 Summary

Mixture-of-Experts (MoE) architectures scale Large Language Models (LLMs) efficiently, but standard fixed top-\(k\) routing activates the exact same number of expert slots for every token, leading to unnecessary computational redundancy. Existing expert-skipping approaches often rely on router confidence, costly calibration datasets, or additional model training, which hinders their ability to accurately estimate an expert's true contribution.

To resolve these limitations, the paper introduces ACE, a training-free, calibration-free, and checkpoint-preserving framework designed for token-adaptive expert skipping in MoE-based LLMs.

⚙️ Core Components of ACE

ACE features two complementary mechanisms: 1. Global Spectral Proxy (GSP): Estimates the global transformation capacity by analyzing the coupled gate, up, and down projections along with RMSNorm scaling. 2. Router-Conditioned Refinement (RCR): Constructs expert-specific direction prototypes leveraging centered router weights, evaluating expert responses along routing-preferred directions.

During inference, ACE integrates both estimates alongside runtime router gates. An expert slot is skipped only when both views independently categorize it as a low-contribution component, while the top-1 expert is always retained. Because all expert statistics are computed offline, online inference requires only efficient table lookups and lightweight scalar operations.

⚙️ Core Components of ACE

ACE features two complementary mechanisms: 1. Global Spectral Proxy (GSP): Estimates the global transformation capacity by analyzing the coupled gate, up, and down projections along with RMSNorm scaling. 2. Router-Conditioned Refinement (RCR): Constructs expert-specific direction prototypes leveraging centered router weights, evaluating expert responses along routing-preferred directions.

During inference, ACE integrates both estimates alongside runtime router gates. An expert slot is skipped only when both views independently categorize it as a low-contribution component, while the top-1 expert is always retained. Because all expert statistics are computed offline, online inference requires only efficient table lookups and lightweight scalar operations.

📊 Experimental Results

Comprehensive evaluations across three MoE-based LLMs and eight diverse benchmarks show that ACE consistently outperforms traditional static and dynamic baselines, with performance advantages growing more pronounced under aggressive expert skipping scenarios:

  • Example Benchmark (Qwen3.6-35B-A3B at a 50% skipping ratio):
  • Reduces WikiText-2 perplexity by 7.96%.
  • Improves average downstream accuracy by 4.15 percentage points over the strongest competing baseline.

📊 Experimental Results

Comprehensive evaluations across three MoE-based LLMs and eight diverse benchmarks show that ACE consistently outperforms traditional static and dynamic baselines, with performance advantages growing more pronounced under aggressive expert skipping scenarios:

  • Example Benchmark (Qwen3.6-35B-A3B at a 50% skipping ratio):
  • Reduces WikiText-2 perplexity by 7.96%.
  • Improves average downstream accuracy by 4.15 percentage points over the strongest competing baseline.

📑 Metadata & References

📑 Metadata & References

license icon