KVBoost：基于偏差引导重计算的块级键值缓存重用技术，助力高效大模型推理
文章背景与核心概要
基于 Transformer 的大语言模型（LLM）在处理请求时，由于需要对键值（KV）张量进行重复计算，往往面临高昂的预填充（prefill）延迟。传统的“前缀缓存”（prefix-caching）系统虽然能缓解这一问题，但其局限性在于必须依赖连续的公共前缀，一旦共享内容出现在提示词的任意位置，这些系统便失效了。
为了突破这一限制，本文提出了 KVBoost，这是一种专为兼容 HuggingFace 的解码器模型设计的块级 KV 缓存重用系统。KVBoost 实现了灵活的缓存重用，无需对基于 RoPE（旋转位置编码）的模型进行任何架构修改，即可在任意位置识别并复用缓存内容。
📌 执行摘要
Transformer-based large language models (LLMs) often suffer from high prefill latency due to the repetitive computation of key-value (KV) tensors for every request. Traditional prefix-caching systems alleviate this cost, but they rely on prompts sharing a contiguous leading prefix, making them ineffective when shared content appears at arbitrary positions.
To overcome this limitation, this paper introduces KVBoost, a chunk-level KV cache reuse system designed for HuggingFace-compatible decoder models. KVBoost enables flexible cache reuse regardless of content position without requiring any architectural modifications to RoPE-based models.
🔑 核心特性与方法论
- Dual-Hash Keying Scheme: Separates positional identity (prefix hash) from content identity (content hash) to facilitate both exact and approximate cache matches.
- Attention Boundary Repair Strategies: To resolve attention boundary errors that arise from independently cached chunks, KVBoost utilizes:
SelectiveRecompute: Re-encodes boundary regions.CacheBlendRecompute: Identifies and recomputes high-deviation tokens following a preliminary probe pass.- Optimization & Resource Management: Features asymmetric KV quantization (
int8/int4), adaptive chunk boundary splitting, and importance-weighted cache eviction under a strictly defined memory budget.
- 双哈希键控方案：将位置标识（前缀哈希）与内容标识（内容哈希）分离，从而促进精确和近似的缓存匹配。
- 注意力边界修复策略：为解决独立缓存块产生的注意力边界错误，KVBoost 采用了：
SelectiveRecompute：对边界区域进行重新编码。CacheBlendRecompute：在初步探测后识别并重新计算高偏差 Token。
- 优化与资源管理：支持非对称 KV 量化（
int8/int4）、自适应块边界分割，以及在严格内存预算下的重要性加权缓存驱逐机制。
📊 评估与结果
Evaluated on the Qwen/Qwen2.5-3B model across 1,000 bug-localization samples, KVBoost demonstrated significant performance gains: * Latency Reduction: Achieved a 4.49× reduction in time-to-first-token (142.4 ms compared to 639.1 ms). * Efficiency: Outperformed standard prefix caching by 16%. * Accuracy: Maintained high accuracy with no performance degradation (99.2% vs. 99.1%).
在 Qwen/Qwen2.5-3B 模型上，针对 1,000 个 Bug 定位样本进行的评估显示，KVBoost 表现出了显著的性能提升： * 延迟降低：首字延迟（Time-to-first-token）降低了 4.49 倍（从 639.1 毫秒缩短至 142.4 毫秒）。 * 效率提升：比标准前缀缓存方案性能提升了 16%。 * 准确性：在保持高准确率的同时，未出现任何性能下降（99.2% 对比 99.1%）。
🔗 全文与资源
