跳转至

文章背景与核心概要

随着大语言模型（LLM）规模的不断扩大，如何在保证精度的同时实现极致的压缩和加速成为了学术界和工业界的研究热点。二值量化（Binary Quantization）在理论上能够带来极高的压缩率和加速比，但传统方法由于无法摆脱高昂的浮点运算开销或运行时的反量化负担，往往难以在实际硬件中完全释放其性能潜力。

为了弥补这一差距，本文提出了 FluxBin，这是一个通过算法与底层算子（Kernel）协同设计来实现灵活、超低比特 LLM 推理的框架。在算法层面，FluxBin 引入了解耦的行列二值分解（Decoupled Row-Column Binary Decomposition）以及 Hessian 引导的显著性感知混合基底（Hessian-guided saliency-aware hybrid bases），在增强模型表征能力的同时保留了关键信息；在算子层面，它通过带尺度融合的查找表构建方法（LUT Building with Scale Fusion）和虚拟列映射（Virtual Columnar Mapping），将不规则的稀疏矩阵转化为高效的稠密执行。实验表明，FluxBin 最高可实现 \(5.92\times\) 的加速和 \(10.19\times\) 的能耗降低，并成功在单张 A100 GPU 上部署 70B 级别的大模型，同时实现 \(4\times\) 的内存占用缩减。

FluxBin: Flexible LUT-based Ultra-low-bit LLM Inference by Algorithm-Kernel Synergy

arXiv ID: arXiv:2608.15602 [cs.LG]
Submitted: August 16, 2026
Authors: Qingyao Yang, Runming Yang, He Xiao, Wendong Xu, Junyu Chen, Haobo Liu, Chenchen Ding, Ruihan Hu, Yik-Chung Wu, Ngai Wong
Links: View PDF | GitHub Repository

📌 Summary

FluxBin is an algorithm-kernel co-design framework designed to achieve flexible, ultra-low-bit Large Language Model (LLM) inference. While binary quantization promises extreme compression and speedups, traditional methods often fail to fully exploit this due to high floating-point arithmetic or runtime dequantization overheads.

FluxBin solves this by synergizing post-training quantization with a highly optimized CUDA kernel. It introduces novel algorithmic components—such as Decoupled Row-Column Binary Decomposition and Hessian-guided saliency-aware hybrid bases—alongside kernel-level optimizations like Lookup Table (LUT) Building with Scale Fusion and Virtual Columnar Mapping. Evaluations show that FluxBin achieves up to a \(5.92\times\) speedup and \(10.19\times\) energy savings while enabling the deployment of 70B-scale models on a single A100 GPU with a \(4\times\) memory reduction.

FluxBin 是一个算法与算子协同设计（Co-design）框架，旨在实现灵活、超低比特的大语言模型（LLM）推理。尽管二值量化在理论上能够带来极端的压缩和加速效果，但传统方法由于持续依赖高昂的浮点算术运算或运行时的反量化开销，往往无法充分挖掘其加速潜力。

FluxBin 通过将训练后量化（PTQ）与高度优化的 CUDA 算子进行协同设计来解决这一问题。在算法层面，它引入了创新的组件——例如解耦的行列二值分解（Decoupled Row-Column Binary Decomposition）和Hessian 引导的显著性感知混合基底（Hessian-guided saliency-aware hybrid bases）；在算子层面，配合了带尺度融合的查找表（LUT）构建方法以及虚拟列映射（Virtual Columnar Mapping）等优化手段。评估表明，FluxBin 最高可实现 \(5.92\times\) 的加速和 \(10.19\times\) 的能耗降低，同时能够将 70B 规模的模型部署在单张 A100 GPU 上，并实现 \(4\times\) 的内存缩减。

🧠 Abstract

While binary quantization theoretically promises extreme compression and acceleration for Large Language Models (LLMs), existing research often overlooks the necessity of specialized hardware kernels, thus failing to unleash the full acceleration potential due to persistent reliance on expensive floating-point arithmetic or runtime dequantization overheads. To bridge this gap, we propose FluxBin (Flexible LUT-based Ultra-low-bit eXecution with Binary bases), an algorithm-kernel co-design that synergizes post-training quantization with a highly optimized CUDA kernel. Algorithmically, we introduce Decoupled Row-Column Binary Decomposition to enhance representational capacity while maintaining hardware efficiency, complemented by a Hessian-guided saliency-aware hybrid bases that preserve critical information. At the kernel level, we implement a Lookup Table Building Approach with Scale Fusion to reduce floating-point arithmetic, featuring a Virtual Columnar Mapping that transforms irregular, sparse, and salient matrices into dense execution. Extensive evaluations demonstrate FluxBin achieves up to \(5.92\times\) speedup and \(10.19\times\) energy savings across diverse model architectures, delivering comparable accuracy to heavily fine-tuned methods. This effectively enables the deployment of 70B-scale models on one single A100 GPU with a \(4\times\) memory reduction.

尽管二值量化在理论上为大语言模型（LLM）提供了极端的压缩和加速前景，但现有研究往往忽视了对专用硬件算子的需求，由于持续依赖昂贵的浮点运算或运行时的反量化开销，从而未能释放全部的加速潜力。为了填补这一空白，我们提出了 FluxBin（基于二值基底的灵活查找表超低比特执行框架，Flexible LUT-based Ultra-low-bit eXecution with Binary bases），这是一个将训练后量化与高度优化的 CUDA 算子相结合的算法-算子协同设计方案。在算法上，我们引入了解耦行列二值分解，在保持硬件效率的同时增强了表征能力，并辅以 Hessian 引导的显著性感知混合基底以保留关键信息。在算子层面，我们实现了带有尺度融合的查找表构建方法以减少浮点算术运算，并采用虚拟列映射将不规则、稀疏且显著的矩阵转换为稠密执行。广泛的评估表明，FluxBin 在各种模型架构中实现了高达 \(5.92\times\) 的加速和 \(10.19\times\) 的能耗降低，并提供了与经过大量微调的方法相当的准确率。这有效地使得在单张 A100 GPU 上部署 70B 规模的模型成为可能，同时减少了 \(4\times\) 的内存占用。

⚙️ Key Innovations

1. Algorithmic Enhancements

  • Decoupled Row-Column Binary Decomposition: Enhances the representational capacity of binary formats without sacrificing hardware efficiency.
  • Hessian-Guided Saliency-Aware Hybrid Bases: Intelligently preserves critical model information by prioritizing salient parameters based on Hessian metrics.

1. 算法增强

  • 解耦行列二值分解（Decoupled Row-Column Binary Decomposition）： 在不牺牲硬件效率的前提下，增强二值格式的表征能力。
  • Hessian 引导的显著性感知混合基底（Hessian-Guided Saliency-Aware Hybrid Bases）： 基于 Hessian 指标优先处理显著参数，智能地保留模型的关键信息。

2. Kernel-Level Optimizations

  • Lookup Table (LUT) Building Approach with Scale Fusion: Minimizes expensive floating-point arithmetic operations during execution.
  • Virtual Columnar Mapping: Dynamically transforms irregular, sparse, and salient weight matrices into highly efficient dense execution layouts.

2. 算子层优化

  • 带尺度融合的查找表（LUT）构建方法： 最大限度地减少执行过程中高昂的浮点算术运算。
  • 虚拟列映射（Virtual Columnar Mapping）： 动态将不规则、稀疏且显著的权重矩阵转换为高效的稠密执行布局。

📊 Performance Highlights

  • Speedup: Up to \(5.92\times\) faster inference.
  • Energy Efficiency: Up to \(10.19\times\) energy savings across varied architectures.
  • Memory Footprint: Achieves a \(4\times\) reduction in memory, allowing 70B-scale LLMs to operate efficiently on a single A100 GPU.

📊 性能亮点

  • 推理加速： 实现最高达 \(5.92\times\) 的推理加速。
  • 能效比： 在不同模型架构中实现最高达 \(10.19\times\) 的能耗节省。
  • 内存占用： 内存占用减少 \(4\times\)，允许 70B 规模的大模型在单张 A100 GPU 上高效运行。

🔗 Additional Resources

🔗 其他资源