跳转至

文章背景与核心概要

混合专家（MoE）大语言模型的参数高效微调（PEFT）传统上采用为每个专家附加独立低秩适配器的方法。然而，这种按专家划分的方法会导致参数碎片化、梯度监督稀疏以及执行效率低下。为了解决这些局限性，本文提出了 ACE（Adapter Consolidation across Experts） 框架。

ACE 在微调过程中识别并对齐功能相似的 LoRA 适配器，在保持相同参数预算的前提下，将其替换为小组共享、更高秩的 LoRA 模块。此外，它还将原本碎片化的计算过程整合为更大规模的组级通用矩阵乘法（GEMM）。在涵盖 12 个数据集和 4 个 MoE 主干网络的评估中，ACE 在不增加峰值内存的前提下，实现了顶尖的准确率，并将实际训练时间加速了 \(1.31\times\) 至 \(1.48\times\)。

ACE: Adapter Consolidation across Experts for Parameter-Efficient Fine-Tuning of MoE LLMs

Summary

混合专家（MoE）模型的参数高效微调（PEFT）传统上需要在每个专家上附加单独的低秩适配器。然而，这种按专家划分的方法会导致碎片化、稀疏的梯度监督以及低效的执行。

Parameter-efficient fine-tuning (PEFT) of Mixture-of-Experts (MoE) models traditionally involves attaching separate low-rank adapters to each expert. However, this expert-wise approach leads to fragmentation, sparse gradient supervision, and inefficient execution.

为了解决这些局限性，本文引入了 ACE（Adapter Consolidation across Experts）。ACE 在微调期间识别并分组功能相似的 LoRA 适配器，在相同的参数预算下用组共享的高秩 LoRA 模块替换它们。此外，它还将碎片化的计算整合为更大的组级通用矩阵乘法（GEMM）。在跨 12 个数据集和四个 MoE 主干网络的评估中，ACE 实现了顶级的准确率，同时在不增加峰值内存使用的情况下，将实际训练时间（wall-clock training time）缩短了 \(1.31\times\) 到 \(1.48\times\)。

To address these limitations, the paper introduces ACE (Adapter Consolidation across Experts). ACE identifies and groups functionally similar LoRA adapters during fine-tuning, replacing them with group-shared, higher-rank LoRA modules under the same parameter budget. Furthermore, it consolidates fragmented computations into larger group-level General Matrix Multiplications (GEMMs). Evaluated across 12 datasets and four MoE backbones, ACE achieves top-tier accuracy while delivering a \(1.31\times\) to \(1.48\times\) speedup in wall-clock training time without increasing peak memory usage.

Paper Metadata

  • arXiv Identifier: arXiv:2609.06072 [cs.LG]
  • Authors: Ahin Lee, Sehyun Yun, Joonha Park, Taesik Gong
  • Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
  • Status: Accepted to EMNLP 2026 (23 pages, 13 figures)
  • License: Creative Commons Attribution 4.0 International <img alt="license icon" role="presentation" src=".././images/345c7ad61f1b.png">

Abstract

混合专家（MoE）模型的参数高效微调（PEFT）通常会为每个专家附加一个单独的低秩适配器。这种按专家设计的架构通过以下三种方式导致了适应过程的碎片化： 1. 容量被分散到狭窄的低秩更新中。 2. 在稀疏路由下，梯度监督变得稀疏且不均衡。 3. 执行过程被分解为许多小的 GEMM。

Parameter-efficient fine-tuning (PEFT) of mixture-of-experts (MoE) models commonly attaches a separate low-rank adapter to each expert. This expert-wise design fragments adaptation in three ways: 1. Capacity is split across narrow low-rank updates. 2. Gradient supervision becomes sparse and imbalanced under sparse routing. 3. Execution is decomposed into many small GEMMs.

我们发现，这种按专家划分的隔离通常是不必要的，因为在微调过程中，LoRA 适配器的子集会在功能上变得相似，这暴露出专家专用适配器之间的冗余性。

We find that such expert-wise separation is often unnecessary, as subsets of LoRA adapters become functionally similar during fine-tuning, revealing redundancy among expert-specific adapters.

基于这种冗余性，我们提出了 ACE（Adapter Consolidation across Experts）。它将冗余的专家进行分组，并在相同的 PEFT 预算下，用组共享的高秩 LoRA 模块替换其专用的适配器。ACE 进一步引入了分组适配器执行机制，将碎片化的按专家计算整合为更少、更大的组级 GEMM。

Based on this redundancy, we propose ACE (Adapter Consolidation across Experts), which groups redundant experts and replaces their expert-specific adapters with group-shared higher-rank LoRA modules under the same PEFT budget. ACE further introduces grouped adapter execution, which consolidates fragmented expert-wise adapter computations into fewer, larger group-level GEMMs.

通过涵盖 12 个数据集和四个 MoE 主干网络的评估，ACE 在具有完整基线覆盖的三种主干网络上，实现了参数匹配的 PEFT 方法中所观察到的最高平均准确率，同时在不增加峰值内存的情况下，比按专家划分的 LoRA 实现了 \(1.31\times\) 至 \(1.48\times\) 的实际训练加速。

Across evaluations covering 12 datasets and four MoE backbones, ACE achieves the highest observed mean accuracy among the parameter-matched PEFT methods on the three backbones with complete baseline coverage, while providing \(1.31\times\) to \(1.48\times\) wall-clock training speedup over expert-wise LoRA without increasing peak memory.