跳转至

面向上下文密集型任务的 KV 缓存卸载技术

文章背景与核心概要

随着长上下文大语言模型（LLM）在各个领域的广泛应用，管理键值（KV）缓存的内存占用和推理延迟变得至关重要。尽管 KV 缓存卸载（KV-cache offloading）作为一种省内存的解决方案被提出，但先前的评估在很大程度上忽略了需要从上下文中提取大量信息密集型任务的场景。

本文针对上下文密集型任务（即模型必须直接从输入提示词中提取海量信息的场景）深入研究了 KV 缓存卸载的效果。为了衡量这一点，作者推出了用于结构化知识提取的 Text2JSON 基准测试。通过在 Llama 3 和 Qwen 3 等模型上评估现代 KV 卸载技术，研究人员发现由低秩键投影和不可靠路标（landmarks）引起的性能显著下降问题。作者提出了一种更简单、替代性的策略，在多个 LLM 家族和基准测试中大幅提升了准确率，这凸显了对长上下文压缩技术进行严格评估的迫切需求。

文档详情 (Document Details)

Document Details

Metadata Details
arXiv ID arXiv:2604.08426
Primary Subject Machine Learning (cs.LG)
Secondary Subjects Artificial Intelligence (cs.AI), Computation and Language (cs.CL)
Authors Andrey Bocharnikov, Ivan Ermakov, Denis Kuznedelev, Vyacheslav Zhdanovskiy, Yegor Yershov
Submission History • v1: Apr 9, 2026
• v2–v4: May 2026
• v5 (Latest): Sep 1, 2026
License Creative Commons Attribution 4.0 International

摘要 (Abstract)

With the growing demand for long-context LLMs across a wide range of applications, the key-value (KV) cache has become a critical bottleneck for both latency and memory usage. Recently, KV-cache offloading has emerged as a promising approach to reduce memory footprint and inference latency while preserving accuracy. Prior evaluations have largely focused on tasks that do not require extracting large amounts of information from the context. In this work, we study KV-cache offloading on context-intensive tasks: problems where the solution requires looking up a lot of information from the input prompt. We create and release the Text2JSON benchmark, a highly context-intensive task that requires extracting structured knowledge from raw text. We evaluate modern KV offloading on Text2JSON and other context-intensive tasks and find significant performance degradation on both Llama 3 and Qwen 3 models. Our analysis identifies two key reasons for poor accuracy: low-rank projection of keys and unreliable landmarks, and proposes a simpler alternative strategy that significantly improves accuracy across multiple LLM families and benchmarks. These findings highlight the need for a comprehensive and rigorous evaluation of long-context compression techniques.

随着各应用领域对长上下文大语言模型（LLM）的需求不断增长，键值（KV）缓存已成为制约推理延迟和内存使用的关键瓶颈。近期，KV 缓存卸载作为一种兼顾准确率、可减少内存占用和推理延迟的有前景的方法崭露头角。先前的评估主要集中在那些不需要从上下文中提取大量信息的任务上。在这项工作中，我们研究了上下文密集型任务中的 KV 缓存卸载：即解决方案需要从输入提示词中检索大量信息的问答场景。我们创建并发布了 Text2JSON 基准测试，这是一项高度依赖上下文的任务，需要从原始文本中提取结构化知识。我们在 Text2JSON 和其他上下文密集型任务上评估了现代 KV 卸载技术，发现 Llama 3 和 Qwen 3 模型均出现了显著的性能退化。我们的分析指出了准确率低下的两个主要原因：键的低秩投影以及不可靠的路标，并提出了一种更简单的替代策略，能够在多个大模型系列和基准测试中显著提升准确率。这些发现凸显了对长上下文压缩技术进行全面、严格评估的必要性。