文章背景与核心概要
混合专家（Mixture-of-Experts, MoE）模型通过引入稀疏激活机制，在不按比例增加每个 Token 计算量的前提下大幅扩展了模型的总参数容量。然而，由于所有专家网络必须同时驻留在内存中，其实际部署依然面临严峻的内存带宽和显存占用挑战。虽然近期的理论研究表明，在微调过程中剪枝那些路由器范数（router-norm）变化最小的专家可以有效保持模型准确率，但该方法此前高度依赖成本高昂的全量微调。
本文旨在探究轻量级的参数高效适应（Parameter-Efficient Adaptation）是否能够成功恢复这一剪枝信号。研究团队通过使用参数高效适配器对模型进行简短的微调，根据诱导出的 \(\ell_2\) 路由器变化对专家进行排序，并单次剪枝变化最小的专家，从而验证了一种实用且可扩展的专家剪枝方法。实验表明，该方法在保持模型性能的同时大幅降低了计算与存储开销，为大规模 MoE 模型的轻量化部署提供了坚实的理论支撑与实践路径。
Router Sensitivity Under Lightweight Fine-Tuning Identifies Prunable Experts in Mixture-of-Experts Models
Router Sensitivity Under Lightweight Fine-Tuning Identifies Prunable Experts in Mixture-of-Experts Models
arXiv: 2608.07890 [cs.LG]
Submitted: August 8, 2026
Authors: Ali Janati, Kaoutar El Maghraoui, Xinyi Luo, Wenyuan Shen, Owen Zou, Yankai Mao
arXiv: 2608.07890 [cs.LG]
Submitted: August 8, 2026
Authors: Ali Janati, Kaoutar El Maghraoui, Xinyi Luo, Wenyuan Shen, Owen Zou, Yankai Mao
📌 Summary
📌 Summary
Mixture-of-Experts (MoE) models scale total parameter capacity without proportionally increasing per-token compute, but deployment remains memory-intensive because every expert must be stored in memory. While recent theory suggests that pruning experts with the smallest router-norm changes during fine-tuning can preserve model accuracy, this approach previously relied on costly full fine-tuning.
Mixture-of-Experts (MoE) models scale total parameter capacity without proportionally increasing per-token compute, but deployment remains memory-intensive because every expert must be stored in memory. While recent theory suggests that pruning experts with the smallest router-norm changes during fine-tuning can preserve model accuracy, this approach previously relied on costly full fine-tuning.
This paper investigates whether lightweight parameter-efficient adaptation can successfully recover this pruning signal. By briefly fine-tuning a model using parameter-efficient adapters, ranking experts by their induced \(\ell_2\) router change, and pruning the least-changed experts in a single shot, the authors demonstrate a practical and scalable expert pruning method: * Efficiency: Router-only LoRA trains just 0.002% of the model parameters while outperforming all-module LoRA configurations. * Performance: On the Mixtral-8×7B-Instruct model, router-guided pruning reduces memory consumption by 49% and per-token latency by 37%, retaining nearly 1.8 times the accuracy of magnitude-based or random pruning at maximal compression. * Generalizability: The criteria transfers effectively to other architectures, such as Qwen1.5-MoE fine-tuned for mathematics, retaining 49.7% mean accuracy across eleven benchmarks even with half of its experts removed.
This paper investigates whether lightweight parameter-efficient adaptation can successfully recover this pruning signal. By briefly fine-tuning a model using parameter-efficient adapters, ranking experts by their induced \(\ell_2\) router change, and pruning the least-changed experts in a single shot, the authors demonstrate a practical and scalable expert pruning method: * Efficiency: Router-only LoRA trains just 0.002% of the model parameters while outperforming all-module LoRA configurations. * Performance: On the Mixtral-8×7B-Instruct model, router-guided pruning reduces memory consumption by 49% and per-token latency by 37%, retaining nearly 1.8 times the accuracy of magnitude-based or random pruning at maximal compression. * Generalizability: The criteria transfers effectively to other architectures, such as Qwen1.5-MoE fine-tuned for mathematics, retaining 49.7% mean accuracy across eleven benchmarks even with half of its experts removed.
🔗 Quick Links
🔗 Quick Links
📄 Abstract
📄 Abstract
Mixture-of-Experts (MoE) models decouple total parameters from per-token compute, but deployment still requires storing every expert. Recent theory shows that pruning experts with the smallest router-norm changes during fine-tuning can preserve accuracy, but assumes full fine-tuning. We test whether lightweight adaptation can recover this signal. We briefly fine-tune with a parameter-efficient adapter, rank experts by the induced \(\ell_2\) router change, and prune the least-changed experts in one shot. On Mixtral-8\(\times\)7B-Instruct (44.83% MMLU-Pro), router-only LoRA trains 0.002% of parameters and outperforms all-module LoRA at matched rank with half the experts removed (27.54% vs. 24.42%); signal quality declines as adaptation spreads to attention and expert weights. Accuracy improves monotonically with LoRA rank, reaching 28.76%. IA3, which leaves router weights frozen, matches direct router adaptation, whereas unconstrained additive adapters degrade the signal. Router-guided MMLU-Pro accuracy decays quasi-linearly rather than collapsing, remains nearly 1.8 times that of magnitude-based or random pruning at maximal compression, and reduces memory by 49% and per-token latency by 37%. At 25% compression, retention is competitive with methods using full activation statistics. The criterion also transfers to Qwen1.5-MoE fine-tuned for mathematics, retaining 49.7% mean accuracy over eleven benchmarks with half the experts removed while random pruning falls to single digits. Router sensitivity under lightweight fine-Timothy therefore makes provably motivated expert pruning practical at scale.
📋 Additional Metadata
📋 Additional Metadata
| Field | Details |
|---|---|
| Primary Subject | Machine Learning (cs.LG) |
| Secondary Subjects | Artificial Intelligence (cs.AI) |
| Document Details | 25 pages, 6 figures, 11 tables |
| License | Creative Commons Attribution 4.0 International  |
Field Details Primary Subject Machine Learning ( cs.LG)Secondary Subjects Artificial Intelligence ( cs.AI)Document Details 25 pages, 6 figures, 11 tables License Creative Commons Attribution 4.0 International