跳转至

文章背景与核心概要

大语言模型（LLM）的大规模部署严重受制于高昂的计算成本、内存带宽瓶颈以及环境开销。传统的压缩方法（如稀疏性、量化和低秩近似）通常是孤立部署的，往往会触及精度与效率的平衡壁垒。本文深入探讨了多维度的模型压缩策略，为实现高效、可扩展且高性能的大模型提供了全新的理论与实践框架。

该论文引入了“压缩三位一体”（Compression Trinity）的概念，这是一个将上述三大支柱联合应用的统一框架：稀疏性用于减少计算量，量化用于最小化内存带宽，低秩近似用于恢复丢失的精度。通过涵盖预训练、微调和后训练压缩的专业方法论，该研究证明了压缩三位一体的联合应用能够打造出高效、可扩展且高性能的大模型，甚至在同等参数预算下能够超越未压缩的稠密模型。

压缩三位一体：探索大语言模型压缩的稀疏性、量化与低秩近似 (Compression Trinity: Exploring Sparsity, Quantization, and Low-Rank Approximations for LLM Compression)

作者： Mohammad Mozaffari
机构： 多伦多大学博士论文 (2026)
主要学科： 人工智能 (cs.AI)
arXiv ID： 2608.24070 | DOI： 10.48550/arXiv.2608.24070
提交时间： 2026年8月25日（156页）

📌 执行摘要 (Executive Summary)

大语言模型（LLM）的大规模部署严重受制于高昂的计算成本、内存带宽瓶颈以及环境开销。传统的压缩方法——例如稀疏性、量化和低秩近似——通常是孤立部署的，往往会触及精度与效率的平衡壁垒。

本论文引入了“压缩三位一体”（"Compression Trinity"），这是一个将这三大支柱联合应用的统一框架： * 稀疏性减少计算量。 * 量化最小化内存带宽。 * 低秩近似恢复丢失的精度。

通过涵盖预训练、微调和后训练压缩的专业方法论，该论文证明了压缩三位一体的联合应用能够实现高度高效、可扩展且高性能的大语言模型，甚至能够超越未压缩的稠密模型。

Deploying Large Language Models (LLMs) at scale is heavily bottlenecked by prohibitive computational, memory bandwidth, and environmental costs. Traditional compression methods—such as sparsity, quantization, and low-rank approximations—are typically deployed in isolation, often hitting an accuracy-efficiency barrier.

This thesis introduces the "Compression Trinity," a unified framework that applies these three pillars jointly: * Sparsity reduces computation. * Quantization minimizes memory bandwidth. * Low-Rank Approximations recover lost accuracy.

Through specialized methodologies spanning pretraining, fine-tuning, and post-training compression, the thesis demonstrates that the joint application of the Compression Trinity enables highly efficient, scalable, and high-performance LLMs that can even outperform uncompressed dense models.

🔬 框架支柱与方法论 (Framework Pillars & Methodologies)

该研究引入了几项基于“压缩三位一体”框架构建的新颖技术：

1. 预训练加速 (Pretraining Acceleration)

  • MKOR： 通过块对角稀疏性和低秩求逆来近似曲率，从而保持量化状态的数值稳定性。它将曲率更新复杂度从 \(\mathcal{O}(d^3)\) 降低到 \(\mathcal{O}(d^2)\)，并将收敛速度比 KFAC 提升高达 1.85倍。
  • SLoPe： 通过针对 \(N:M\) 稀疏性的双重剪枝反向传播，将训练速度提升高达 1.25倍，并在训练的最后 1% 阶段采用低秩“惰性”适配器来恢复精度。

The research introduces several novel techniques built upon the Compression Trinity framework:

1. Pretraining Acceleration

  • MKOR: Approximates curvature via block-diagonal sparsity and low-rank inversion to maintain numerical stability for quantized states. It reduces curvature update complexity from \(\mathcal{O}(d^3)\) to \(\mathcal{O}(d^2)\) and accelerates convergence by up to 1.85× over KFAC.
  • SLoPe: Accelerates training by up to 1.25× via a double-pruned backward pass for \(N:M\) sparsity, employing low-rank "lazy" adapters in the final 1% of training to recover accuracy.

2. 后训练与微调压缩 (Post-Training & Fine-Tuning Compression)

  • OPTIMA： 通过将权重重建表述为全局最优的列向二次规划，在零训练机制下稳定静态掩码，将零样本准确率提升高达 3.97%。
  • PATCH： 在给定微调预算的情况下，通过学习 0% 到 50% 之间的动态混合稀疏度比例，突破了静态掩码的局限性，带来了高达 1.38倍 的加速。
  • SLiM： 一次性实现完整的压缩三位一体。它利用数学推导出的低秩适配器来恢复因量化和稀疏性而丢失的信息，相比最先进的方法将准确率提升了高达 5.66%，在同等参数预算下比未压缩的稠密模型表现优异 0.6%。

2. Post-Training & Fine-Tuning Compression

  • OPTIMA: Stabilizes static masks in a zero-training regime by formulating weight reconstruction as globally optimal column-wise quadratic programs, improving zero-shot accuracy by up to 3.97%.
  • PATCH: Breaks the limitations of static masks given a fine-tuning budget by learning a dynamic hybrid sparsity ratio between 0% and 50%, yielding up to 1.38× speedups.
  • SLiM: Realizes the full Compression Trinity in a single shot. It utilizes mathematically derived low-rank adapters to recover information lost due to quantization and sparsity, improving accuracy by up to 5.66% over state-of-the-art methods and outperforming uncompressed dense models by 0.6% at equal parameter budgets.

本论文的各章节延伸了以下基础工作： * MKOR： arXiv:2306.01685 * SLoPe： arXiv:2405.16325 * OPTIMA： arXiv:2512.13886 * PATCH： arXiv:2509.23410 * SLiM： arXiv:2410.09615

官方机构记录可通过 多伦多大学 Scholaris 获取。

The chapters of this thesis extend the following foundational works: * MKOR: arXiv:2306.01685 * SLoPe: arXiv:2405.16325 * OPTIMA: arXiv:2512.13886 * PATCH: arXiv:2509.23410 * SLiM: arXiv:2410.09615

Official institutional record available via University of Toronto Scholaris.

(注意：保留了用于 UI 渲染一致性的许可证图标资产：license icon)

(Note: License icon asset preserved for UI rendering parity: license icon)