超全训练，紧凑部署：扩展结构化大语言模型剪枝的恢复能力

文章背景与核心概要

尽管大型语言模型（LLM）在各类任务中取得了最先进的性能，但由于高内存、低延迟和高能耗的要求，其部署成本依然高昂。结构化剪枝通过移除整个架构组件来解决这些问题；然而，恢复阶段往往受到恢复模块的容量与被移除知识的复杂性之间不匹配的瓶颈限制——作者将这一局限性称为容量-知识不对称（capacity-knowledge asymmetry）。

为了解决这一问题，研究人员推出了 OverRep（过完备重参数化，Overcomplete Reparameterization）框架，其核心原则为：“超全训练，紧凑部署”。该框架通过在训练期间临时过参数化恢复模块以有效吸收从原模型中蒸馏出的复杂知识，并在恢复后通过精确的代数合并将过完备重参数化合并为数学上等效的紧凑模块，从而在推理时的架构或计算成本上实现零开销。在三个主流模型家族上的评估表明，在保持相当内存使用和 TFLOPs 的同时，OverRep 在 25% 剪枝率下将保留的推理性能提升了高达 5.5 个百分点，在 50% 剪枝率下提升了高达 8.4 个百分点。

训练超全，部署紧凑：扩展结构化大语言模型剪枝的恢复能力 (Train Overcomplete, Deploy Compact: Scaling Recovery Capacity for Structured LLM Pruning)

arXiv ID: arXiv:2609.06974 [cs.CL]
Authors: Seungmin Oh, Donggeon Lee, Jongbin Ryu
Submitted: September 7, 2026
Venue: Accepted by the Conference on Empirical Methods in Natural Language Processing (EMNLP) 2026
Links: View PDF | HTML Version | GitHub Repository

执行摘要 (Executive Summary)

While large language models (LLMs) achieve state-of-the-art performance across a wide variety of tasks, deploying them remains expensive due to high memory, latency, and energy requirements. Structured pruning addresses these issues by removing entire architectural components; however, the recovery stage is often bottlenecked by a mismatch between the capacity of the recovery module and the complexity of the removed knowledge—a limitation the authors term capacity-knowledge asymmetry.

尽管大型语言模型（LLM）在各种任务中都取得了最先进的性能，但由于高内存、延迟和能源需求，部署它们仍然很昂贵。结构化 pruning（剪枝）通过移除整个架构组件来解决这些问题；然而，恢复阶段往往受到恢复模块的容量与被移除知识的复杂性之间不匹配的瓶颈限制——作者将这种局限性称为容量-知识不对称。

To solve this, the researchers introduce OverRep, an Overcomplete Reparameterization framework guided by the principle: "Train overcomplete, deploy compact."

为了解决这个问题，研究人员引入了 OverRep，这是一个由以下原则指导的过完备重参数化（Overcomplete Reparameterization）框架：“训练超全，部署紧凑。”

Key Innovations:

Overparameterized Recovery: Temporarily overparameterizes the recovery module during training to effectively absorb complex knowledge distilled from the original model.

Exact Algebraic Merging: Post-recovery, the overcomplete re-parameterization is algebraically merged into a mathematically equivalent compact module, ensuring zero overhead to the inference-time architecture or computational cost.

Annealed Activation: Enables nonlinear training dynamics while smoothly converging to a linear regime required for exact algebraic merging.

Substantial Performance Gains: Evaluated across three backbone families, OverRep improves retained reasoning performance over strong recovery baselines by up to 5.5 points at 25% pruning and 8.4 points at 50% pruning, while maintaining comparable memory usage and TFLOPs.

关键创新：

过参数化恢复： 在训练期间临时对恢复模块进行过参数化，以有效吸收从原始模型中蒸馏出的复杂知识。

精确代数合并： 恢复后，过完备重参数化在代数上被合并为一个数学上等效的紧凑模块，确保推理时的架构或计算成本零开销。

退火激活： 启用非线性训练动态，同时平滑收敛到精确代数合并所需的线性状态。

显着的性能提升： 在三个骨干模型系列上进行评估，与强大的恢复基线相比，OverRep 将保留的推理性能在 25% 剪枝时提升高达 5.5 个百分点，在 50% 剪枝时提升高达 8.4 个百分点，同时保持可比的内存使用量和 TFLOPs。

元数据与参考信息 (Metadata & Reference Information)

Primary Subject: Computation and Language (cs.CL)
Secondary Subjects: Artificial Intelligence (cs.AI)
DOI: 10.48550/arXiv.2609.06974

Citation:

@misc{oh2026trainovercompletedeploycompact,
      title={Train Overcomplete, Deploy Compact: Scaling Recovery Capacity for Structured LLM Pruning}, 
      author={Seungmin Oh and Donggeon Lee and Jongbin Ryu},
      year={2026},
      eprint={2609.06974},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

主要主题： 计算与语言（cs.CL）
次要主题： 人工智能（cs.AI）
DOI： 10.48550/arXiv.2609.06974

引用：

@misc{oh2026trainovercompletedeploycompact,
      title={Train Overcomplete, Deploy Compact: Scaling Recovery Capacity for Structured LLM Pruning}, 
      author={Seungmin Oh and Donggeon Lee and Jongbin Ryu},
      year={2026},
      eprint={2609.06974},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}