跳转至

文章背景与核心概要

在长上下文大型语言模型（LLM）的解码阶段，键值（KV）缓存会带来严重的内存带宽瓶颈。虽然将量化的KV缓存存储在密集的片上非易失性内存（NVM）中可以消除片外数据传输，但现有的以GPU为中心的量化方法（如KIVI和KVQuant）无法与NVM硬件约束进行高效匹配，从而导致较高的元数据开销或不兼容的异常值处理。

本文引入了一种专为密集片上NVM架构协同设计的接口感知KV缓存量化方案。通过利用随机旋转和逐向量归一化，该方法统一了坐标范围，使得单个固定的码本即可服务于每个张量类型的所有KV缓存。该设计实现了显著的硬件效率——与基准NVM映射相比，KV读取能耗降低了 3.1–3.6 倍，且元数据开销比 KIVI 低 8 倍，同时在参数量从 3B 到 14B、上下文长度高达 32k 的模型中保持了稳健的准确性。

Interface-Aware KV Cache Quantization for Dense On-Chip NVM in Long-Context LLM Decoding

Summary

The key-value (KV) cache creates a severe memory bandwidth bottleneck during long-context large language model (LLM) decoding. While storing a quantized KV cache in dense on-chip non-volatile memory (NVM) eliminates off-chip data transfers, existing GPU-centric quantization methods (such as KIVI and KVQuant) fail to align efficiently with NVM hardware constraints, suffering from high metadata overhead or incompatible outlier handling.

This paper introduces an interface-aware KV quantization scheme co-designed specifically for dense on-chip NVM architectures. By utilizing randomized rotation and per-vector normalization, the method standardizes coordinate ranges so that a single fixed codebook serves the entire KV cache per tensor type. This design achieves significant hardware efficiency—yielding 3.1–3.6× lower KV read energy compared to baseline NVM mapping and 8× lower metadata overhead than KIVI—while maintaining robust accuracy across models ranging from 3B to 14B parameters up to 32k context lengths.

Metadata

  • arXiv ID: arXiv:2609.05764 [cs.AR]
  • Authors: Jiahao Zheng, Yifan Qin, Xiaobo Sharon Hu, Yiyu Shi
  • Primary Subject: Hardware Architecture (cs.AR)
  • Conference: IEEE/ACM International Conference on Computer-Aided Design (ICCAD 2026)
  • Submitted Date: September 4, 2026
  • Related DOI: 10.1145/3831252.3845860

Abstract & Key Contributions

  • Problem Addressed: Standard GPU-tailored KV cache quantization techniques incur heavy metadata overhead (e.g., ~25% for KIVI) or require sparse outlier handling incompatible with dense on-chip NVM hardware arrays.
  • Proposed Hardware-Software Co-Design:
  • Stores the quantized KV cache directly in dense on-chip NVM.
  • Employs a small static analog crossbar exclusively for fixed rotation, leaving attention operations in on-chip digital logic.
  • Applies randomized rotation and per-vector normalization to unify coordinate ranges.
  • Performance Highlights:
  • 3.1–3.6× lower KV read energy compared to existing methods mapped to the same NVM.
  • 8× lower metadata overhead relative to KIVI (requiring only one scalar per vector, roughly 3%).
  • Maintains high accuracy for 4-bit KV caches across 3B to 14B models and context lengths up to 32,000 tokens under realistic storage and crossbar noise levels.