文章背景与核心概要
在大语言模型（LLM）的部署中，KV缓存（KV-cache）量化技术长期依赖于离线基准测试的平均表现，这导致实际运行系统无法感知压缩操作是否正在损害当前正在处理的具体请求。为了解决这一痛点，Metask Lab 的研究团队推出了 WitCert。它引入了一种经过数学证明可靠的运行时度量机制——类似于 KV 量化的“DTrace”——能够针对每个（层, 注意力头, 推理步）提供精确注意力与压缩注意力之间总变差（Total Variation）的上界。
该系统采用了双层运行架构：一是确定性的“频带-范数-证人”上界（Band-Norm-Witness Bound），对任意缓存保留的黑盒量化器和任意查询都具有鲁棒性；二是概率型证书（Probabilistic Certificate），专为受控的减法抖动 INT8 量化器设计，并在明确的请求级失败预算下提供更紧凑的界限（其核心定理已在 Lean 4 中完成机器辅助验证）。实验表明，WitCert 能够无缝集成至 SGLang 中，在不增加人工基准测试开销的前提下，将硬 RULER 任务的性能从 22.8 恢复至 79.7，并在相同的内存约束下使 SGLang 的 KV 令牌吞吐量提升了 \(1.88\times\)。
WitCert: Sound Runtime Risk Observability and Gating for KV-Cache Quantization
Authors: Fanzhe Wei, Li Liu, Ziyang Wang, Chenyu Wang (Metask Lab)
Subjects: Hardware Architecture (cs.AR); Artificial Intelligence (cs.AI)
arXiv: 2607.28699 | DOI: 10.48550/arXiv.2607.28699
Submitted: July 30, 2026 (Last revised August 6, 2026)
Artifacts & Code: GitHub Repository
Authors: Fanzhe Wei, Li Liu, Ziyang Wang, Chenyu Wang (Metask Lab)
Subjects: Hardware Architecture (cs.AR); Artificial Intelligence (cs.AI)
arXiv: 2607.28699 | DOI: 10.48550/arXiv.2607.28699
Submitted: July 30, 2026 (Last revised August 6, 2026)
Artifacts & Code: GitHub Repository
Executive Summary
KV-cache quantization has historically relied on offline benchmark averages, leaving deployed systems blind to whether compression is actively degrading the specific request currently being processed. WitCert introduces a provably sound runtime meter—functioning as a "DTrace for KV quantization"—that provides a per-(layer, head, step) upper bound on the total variation between exact and compressed attention.
The system operates via a dual-tier approach: 1. Deterministic Band-Norm-Witness Bound: Sound for any cache-preserving black-box quantizer and arbitrary queries (adaptive-safe, utilizing worst-case Cauchy-Schwarz plus RoPE band-unitarity). 2. Probabilistic Certificate: A tighter bound designed for controlled subtractively-dithered INT8 quantizers under an explicit request-level failure budget (with core theorems machine-checked in Lean 4).
KV-cache quantization has historically relied on offline benchmark averages, leaving deployed systems blind to whether compression is actively degrading the specific request currently being processed. WitCert introduces a provably sound runtime meter—functioning as a "DTrace for KV quantization"—that provides a per-(layer, head, step) upper bound on the total variation between exact and compressed attention.
The system operates via a dual-tier approach: 1. Deterministic Band-Norm-Witness Bound: Sound for any cache-preserving black-box quantizer and arbitrary queries (adaptive-safe, utilizing worst-case Cauchy-Schwarz plus RoPE band-unitarity). 2. Probabilistic Certificate: A tighter bound designed for controlled subtractively-dithered INT8 quantizers under an explicit request-level failure budget (with core theorems machine-checked in Lean 4).
Key Contributions & Findings
1. Real-Time Observability
WitCert integrates seamlessly into SGLang via an environment-guarded patch. Any quantization scheme registered as a single tensor function can be dynamically measured during live serving without manual benchmarking overhead.
1. Real-Time Observability
WitCert integrates seamlessly into SGLang via an environment-guarded patch. Any quantization scheme registered as a single tensor function can be dynamically measured during live serving without manual benchmarking overhead.
2. Meter-Driven Gating & Repair
By combining risk-ranked mitigation (where the witness is saturated) with certified guarantees (where it is informative), WitCert empirically restores model quality floors at benchmark scale. For instance: * Hard RULER tasks recover raw-cast FP8 performance from 22.8 back to 79.7. * Paired-test evaluation bounds the performance drop compared to uncompressed models tightly between \([+0.0, +0.8]\).
2. Meter-Driven Gating & Repair
By combining risk-ranked mitigation (where the witness is saturated) with certified guarantees (where it is informative), WitCert empirically restores model quality floors at benchmark scale. For instance: * Hard RULER tasks recover raw-cast FP8 performance from 22.8 back to 79.7. * Paired-test evaluation bounds the performance drop compared to uncompressed models tightly between \([+0.0, +0.8]\).
3. Cross-Layer Error Cancellation Analysis
The study reveals that aggressive quantization schemes survive thanks to cross-layer error cancellation rather than strict per-step fidelity. In a 28-layer model sweep, no single layer's pollution alone caused output failures (0/28). Furthermore, utilizing the certified INT8 cache enabled SGLang to serve \(1.88\times\) more KV tokens under identical memory constraints.
3. Cross-Layer Error Cancellation Analysis
The study reveals that aggressive quantization schemes survive thanks to cross-layer error cancellation rather than strict per-step fidelity. In a 28-layer model sweep, no single layer's pollution alone caused output failures (0/28). Furthermore, utilizing the certified INT8 cache enabled SGLang to serve \(1.88\times\) more KV tokens under identical memory constraints.
Submission History
- v1: Thu, 30 Jul 2026 11:04:45 UTC (221 KB)
- v2: Thu, 6 Aug 2026 10:44:15 UTC (218 KB)
Submission History
- v1: Thu, 30 Jul 2026 11:04:45 UTC (221 KB)
- v2: Thu, 6 Aug 2026 10:44:15 UTC (218 KB)