基于输出重要性的残差稀疏化：用于压缩专家混合大语言模型

文章背景与核心概要

专家混合（MoE）架构使大语言模型（LLMs）能够实现高效扩展，但也带来了巨大的GPU显存占用，因此迫切需要模型压缩技术。残差稀疏化是一种主流的压缩方法，它将专家的投影矩阵分解为一个共享基础矩阵和每个专家独有的残差矩阵，并对残差进行独立压缩。然而，作者证明了最小化孤立的矩阵误差与保持下游模型准确性之间存在根本性的目标错位。

由于专家输出依赖于多个投影和隐藏表示之间的耦合计算，微小的局部误差可能会级联并导致显著的输出退化。为了解决这一问题，本文推出了 PARSER，一种专注于通过“输出重要性”来保持专家输出误差的新型残差稀疏化方法。实验结果表明，与未压缩模型相比，PARSER显著缩小了准确率差距——在 Qwen 上实现了 1.41\(\times\) 的提升，在 DeepSeek 上实现了 1.44\(\times\) 的提升——同时达到了相同的峰值显存缩减效果。

元数据 (Metadata)

arXiv ID: arXiv:2609.00575 [cs.AI]
作者: Seungwoo Jung, Dohyeok Kwon, Seungmin Cha, Junseok Lee, Yeonho Yoo, Chuck Yoo, Gyeongsik Yang
提交时间: 2026年9月1日
录用会议: EMNLP 2026 (主会)
链接: 查看 PDF, HTML 版本, DOI

执行摘要 (Executive Summary)

专家混合 (MoE) 架构使得大语言模型能够高效扩展，但它们也需要海量的 GPU 显存。为了应对这种显存需求，模型通常会被压缩以减小其内存占用。残差稀疏化是一种具有代表性的压缩技术，它将专家的每个投影矩阵分解为一个共享的基础矩阵和每个专家独有的残差矩阵，然后对残差进行压缩。

然而，作者证明了最小化孤立的矩阵误差与保持压缩后的模型准确率之间存在目标错位。由于专家最终的输出是通过跨多个投影和隐藏表示耦合的计算产生的，因此即使是单个矩阵中的微小误差，也会通过隐藏表示和投影交互进行传播，从而导致巨大的专家输出误差和准确率下降。为了解决这种目标错位，本文提出了 PARSER，这是一种新的残差稀疏化方法，它将压缩目标从最小化孤立的矩阵误差转变为保持专家输出误差。PARSER 通过引入输出重要性（衡量对专家输出误差的实际贡献）来实现这一点。实验结果表明，与现有方法相比，PARSER 将与未压缩模型的准确率差距在 Qwen 上缩小了 1.41\(\times\)，在 DeepSeek 上缩小了 1.44\(\times\)，同时达到了相同的峰值显存缩减效果。

Mixture-of-experts (MoE) architectures scale large language models efficiently, but they demand massive GPU memory. To cope with such demand, models are commonly compressed to reduce their memory footprint. Residual sparsification is a representative compression technique that decomposes each projection matrix of an expert into a shared base matrix and per-expert residual matrix, and then compresses the residuals. Existing sparsification methods compress each residual matrix independently by minimizing its compression error, thereby minimizing the error of each projection matrix. However, our analysis shows that this objective is misaligned with preserving model accuracy after compression. In an expert, the final output is produced through computations coupled across multiple projections and hidden representations. Therefore, even small errors in individual matrices can propagate through hidden representations and projection interactions, leading to large expert output errors and accuracy degradation. To address this misalignment, we propose PARSER, a new residual sparsification method that shifts the compression objective from minimizing isolated matrix errors to preserving the expert output error. PARSER achieves this by introducing output importance, which measures the actual contribution to the expert output error. Our experiments show that, compared with existing methods, PARSER narrows the accuracy gap to the uncompressed model by 1.41\(\times\) on Qwen and 1.44\(\times\) on DeepSeek, while achieving the same peak memory reduction.

摘要 (Abstract)

Mixture-of-experts (MoE) architectures scale large language models efficiently, but they demand massive GPU memory. To cope with such demand, models are commonly compressed to reduce their memory footprint. Residual sparsification is a representative compression technique that decomposes each projection matrix of an expert into a shared base matrix and per-expert residual matrix, and then compresses the residuals. Existing sparsification methods compress each residual matrix independently by minimizing its compression error, thereby minimizing the error of each projection matrix. However, our analysis shows that this objective is misaligned with preserving model accuracy after compression. In an expert, the final output is produced through computations coupled across multiple projections and hidden representations. Therefore, even small errors in individual matrices can propagate through hidden representations and projection interactions, leading to large expert output errors and accuracy degradation. To address this misalignment, we propose PARSER, a new residual sparsification method that shifts the compression objective from minimizing isolated matrix errors to preserving the expert output error. PARSER achieves this by introducing output importance, which measures the actual contribution to the expert output error. Our experiments show that, compared with existing methods, PARSER narrows the accuracy gap to the uncompressed model by 1.41\(\times\) on Qwen and 1.44\(\times\) on DeepSeek, while achieving the same peak memory reduction.

核心贡献与方法亮点 (Key Contributions & Methodological Highlights)

目标错位的识别： 强调了传统残差稀疏化方法的错误所在，即它们将矩阵压缩视为独立任务，而没有考虑到跨隐藏表示和耦合投影的误差传播。
PARSER 框架： 提出了一种优化目标的转变，不再将最小化孤立矩阵误差作为目标，而是直接保持专家输出的完整性。
输出重要性指标： 引入了一种新颖的度量标准，用于量化单个组件对总体专家输出误差的实际贡献。
实证验证： 证明了在相同的峰值显存占用下，该方法在先进的 MoE 架构（Qwen 和 DeepSeek）上具有更优越的压缩质量。

Identification of Objective Misalignment: Highlights how traditional residual sparsification methods err by treating matrix compression independently rather than accounting for error propagation across hidden representations and coupled projections.

PARSER Framework: Proposes a shift in the optimization objective away from isolated matrix error minimization to directly preserving expert output integrity.

Output Importance Metric: Introduces a novel measure quantifying the actual contribution of individual components to the aggregate expert output error.

Empirical Validation: Demonstrates superior compression quality on state-of-the-art MoE architectures (Qwen and DeepSeek) at identical peak memory footprints.