跳转至

全为了1比特：迈向大语言模型真正的1比特训练后量化

文章背景与核心概要

大语言模型（LLM）虽然性能卓越，但其庞大的内存和存储需求使得高效部署变得异常困难。尽管权重二值化是一个很有前景的解决方案，但现有的基于二值化的训练后量化（PTQ）技术由于隐藏的开销，往往会超过名义上的 1 比特存储限制。

为了克服这一局限性，作者推出了 All for 1-Bit (AF1)，这是一个针对大语言模型的真正的 1 比特 PTQ 框架，旨在严格保持在 1.0-BPW（每权重比特数）的预算之内，同时不牺牲模型精度。在 LLaMA、Qwen 和 Gemma 模型系列上的评估表明，AF1 在困惑度和零样本准确率方面均优于现有的二值化方法，与 BF16 相比，平均实现了 2.5 倍的推理加速并减少了 90% 以上的内存占用。

Summary

Large language models (LLMs) offer high performance, but their massive memory and storage demands make efficient deployment difficult. While weight binarization is a promising solution, existing binarization-based Post-Training Quantization (PTQ) techniques often exceed nominal 1-bit storage limits due to hidden overhead.

To overcome this, the authors introduce All for 1-Bit (AF1), a genuine 1-bit PTQ framework for LLMs designed to stay strictly under a 1.0-BPW (bits-per-weight) budget without sacrificing model accuracy. Evaluations across the LLaMA, Qwen, and Gemma model families demonstrate that AF1 outperforms existing binarization methods in perplexity and zero-shot accuracy, delivering an average 2.5x inference speedup and over 90% memory reduction compared to BF16.

论文元数据

Paper Metadata

  • arXiv ID: arXiv:2609.06161 [cs.LG]
  • 研究领域 (Subjects): Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
  • 录用会议 (Accepted At): EMNLP-Main 2026
  • 提交日期 (Submission Date): 2026年9月5日
  • 作者 (Authors): Zhixiong Zhao, Zukang Xu, Guangyu Sun, Lifeng Liu, Dawei Yang
  • 代码仓库 (Code Repository): GitHub - Kishon-zzx/AF1
  • arXiv ID: arXiv:2609.06161 [cs.LG]
  • Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
  • Accepted At: EMNLP-Main 2026
  • Submission Date: September 5, 2026
  • Authors: Zhixiong Zhao, Zukang Xu, Guangyu Sun, Lifeng Liu, Dawei Yang
  • Code Repository: GitHub - Kishon-zzx/AF1

核心框架：AF1

AF1 框架由两个互补的组件组成，旨在保持模型精度的同时强制执行严格的 1 比特预算：

  1. 零空间感知二值分解（NABF）： 通过以下方式增强二值重建：
  2. 海森矩阵感知（Hessian-aware）代理重新参数化
  3. 零空间感知二值分解

  4. 纯缩放全局重建（Scale-only global reconstruction）

  5. 分层夏普利分配（HiSA）： 利用分层夏普利敏感度有效地分配结构容量，以在严苛的约束下保持稳定的性能。

Core Framework: AF1

The AF1 framework consists of two complementary components designed to preserve model accuracy and enforce a strict 1-bit budget:

  1. Null-space-Aware Binary Factorization (NABF): Enhances binary reconstruction through:
  2. Hessian-aware surrogate reparameterization
  3. Null-space-aware binary factorization

  4. Scale-only global reconstruction

  5. Hierarchical Shapley Allocation (HiSA): Assigns structural capacity effectively using hierarchical Shapley sensitivity to maintain stable performance under tight constraints.

链接与资源

license icon license icon