跳转至

准确率并非全部：基于散度的方法评估量化大语言模型的保真度损失

文章背景与核心概要

评估量化大语言模型（LLM）目前在很大程度上依赖于零样本任务准确率（zero-shot task accuracy），该指标仅考虑 argmax 预测结果，而忽略了底层预测分布的变化。这种方法往往会导致不稳定且非单调的行为，从而掩盖了相对于未压缩基础模型（如 BFloat16）的关键保真度损失。

为了解决这一局限性，作者引入了一个对分布敏感的评估框架，该框架通过测量 Token 决策边界处全词表预测分布之间的统计散度来量化信息损失。该研究利用 詹森-香农散度（Jensen-Shannon Divergence） 和 全变差距离（Total Variation Distance） 等指标，结合跨越五种基础架构和四个推理基准的强大实验矩阵，证明了分布感知评估能够为受内存限制的部署提供更可靠的保真度信号。

license icon

Summary

Evaluating quantized Large Language Models (LLMs) heavily relies on zero-shot task accuracy, which only considers argmax predictions and ignores shifts in the underlying predictive distribution. This approach often leads to unstable and non-monotonic behavior, masking critical fidelity losses relative to uncompressed base models (like BFloat16).

To address this limitation, the authors introduce a distribution-sensitive evaluation framework that measures information loss as the statistical divergence between full-vocabulary predictive distributions at the token decision boundary. Utilizing metrics like Jensen-Shannon Divergence and Total Variation Distance, alongside a robust experimental matrix across five foundation architectures and four reasoning benchmarks, the study demonstrates that distribution-aware evaluation provides a much more reliable fidelity signal for memory-constrained deployments.

Metadata

  • arXiv ID: arXiv:2609.07664 [cs.LG]
  • Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
  • Submission Date: September 7, 2026
  • DOI: 10.48550/arXiv.2609.07664

Authors

  • Shahzeb Qamar
  • Lorenz Sparrenberg
  • Christian Bauckhage
  • Baha Rababah
  • Carson Leung
  • Murat Kantarcioglu
  • Cuneyt Gurcan Akcora
  • Rafet Sifa

Abstract

在内存受限的边缘设备上部署大语言模型（LLM）在很大程度上依赖于激进的训练后量化。然而，这些模型的评估在很大程度上基于零样本任务准确率，该准确率仅取决于 argmax 预测，对底层预测分布的变化不敏感。因此，在渐进式量化下，准确率可能会表现出不稳定、非单调的行为，掩盖了相对于 BFloat16（BF16）未压缩基础模型的实质性保真度损失，并提供具有误导性的部署信号。

我们引入了一个对分布敏感的评估框架，将量化 LLM 中的信息损失量化为 Token 决策边界处全词表预测分布之间的散度。我们计算了全精度模型和量化模型输出之间的统计距离（包括詹森-香农散度和全变差距离），从而能够对分布偏移进行细粒度分析。利用该框架，我们量化了相对于 BF16 参考基准的概率质量位移和分布漂移，捕捉了未在 Top-1 准确率中反映出来的预测分布变化。

我们在渐进式量化方案（从未压缩的 BF16 到 Q2_K）下，跨越五种基础架构和四个推理基准进行了包含 120 次运行的实验矩阵测试，从而提供了系统的保真度分析。我们的结果表明，在更强的量化下，散度指标通常会增加，这为主任务准确率补充了一个保真度信号。在测试的 llama.cpp 方案中，在相似的内存占用下，混合精度 Q4_K 通常比统一的 Q4_0 产生更低的散度。这些发现促使分布感知评估成为任务准确率的实用诊断补充；它们并未直接确立正确性、校准性、安全性或用户感知的质量。

Deployment of Large Language Models (LLMs) on memory-constrained edge devices relies heavily on aggressive post-training quantization. However, evaluating these models is largely based on zero-shot task accuracy, which depends solely on argmax predictions and is insensitive to changes in the underlying predictive distribution. Consequently, accuracy can exhibit unstable, non-monotonic behavior under progressive quantization, masking substantial fidelity loss relative to the BFloat16 (BF16) uncompressed base model and providing misleading deployment signals.

We introduce a distribution-sensitive evaluation framework quantifying information loss in quantized LLMs as the divergence between full-vocabulary predictive distributions at the token decision boundary. We compute statistical distances, including Jensen-Shannon Divergence and Total Variation Distance, between outputs of full-precision and quantized models, enabling a fine-grained analysis of distributional shift. Using this framework, we quantify probability mass displacement and distributional drift relative to the BF16 reference, capturing predictive distribution changes not reflected in top-1 accuracy.

We conduct a 120-run experimental matrix across five foundation architectures and four reasoning benchmarks under progressive quantization regimes, from uncompressed BF16 to Q2_K, providing a systematic fidelity analysis. Our results show divergence metrics generally increase under stronger quantization, complementing task accuracy with a fidelity signal. Across tested llama.cpp schemes, mixed-precision Q4_K generally yields lower divergence than uniform Q4_0 at similar memory footprints. These findings motivate distribution-aware evaluation as a practical diagnostic complement to task accuracy; they do not directly establish correctness, calibration, safety, or user-perceived quality.