跳转至

文章背景与核心概要

在大模型处理长文本时，虽然模型绝大部分的注意力只集中在上下文的一小部分上，但传统方法仍需要扫描完整的键值（KV）缓存来寻找相关内容。这导致在生成回复的每一步都会产生巨大的计算开销。现有的外部代理评分机制虽能缓解这一问题，但仍无法消除每步 \(\mathcal{O}(N)\) 的计算成本。

本文提出了一种名为“声明式注意力（Declarative Attention, DA）”的创新方法。该方法基于一个直觉：模型自身应该最清楚上下文的哪些部分是相关的。通过让模型在思维链中直接声明需要关注的位置，推理引擎可以像处理工具调用一样解析这些声明，从而跳过绝大部分无用的 KV 缓存读取。实验表明，该方法在多个长文本任务中显著减少了解码过程中的注意力标记数量，为实现高效的稀疏注意力开辟了新途径。

Language Models Can Control Their Own Attention

📌 Executive Summary

arXiv:2609.02737 [cs.CL]
Submitted on 2 September 2026
Authors: Namgyu Ho, Huzama Ahmad, Woosung Koh, Se-Young Yun, Tal Schuster, Cicero Nogueira dos Santos

📌 Executive Summary

arXiv:2609.02737 [cs.CL]
Submitted on 2 September 2026
Authors: Namgyu Ho, Huzama Ahmad, Woosung Koh, Se-Young Yun, Tal Schuster, Cicero Nogueira dos Santos

🔍 Abstract & Overview

While language models spend the majority of their attention on only a small fraction of the context, they traditionally read the entire Key-Value (KV) cache to identify the few relevant tokens. For example, if a user queries a previous detail buried inside a 1-million-token conversation, global attention layers are forced to scan the full context for every single token generated in the reply.

Existing solutions attempt to mitigate this cost by pre-selecting relevant tokens using lightweight proxy scores, but this extrinsic scoring still incurs an \(\mathcal{O}(N)\) computational overhead per step.

This paper introduces an intrinsic approach based on a fundamental premise: Shouldn't the model already know which parts of the context are relevant?

To address this, the authors propose Declarative Attention (DA), a protocol that enables language models to declare where they need to attend directly within their chain-of-thought.

🔍 Abstract & Overview

While language models spend the majority of their attention on only a small fraction of the context, they traditionally read the entire Key-Value (KV) cache to identify the few relevant tokens. For example, if a user queries a previous detail buried inside a 1-million-token conversation, global attention layers are forced to scan the full context for every single token generated in the reply.

Existing solutions attempt to mitigate this cost by pre-selecting relevant tokens using lightweight proxy scores, but this extrinsic scoring still incurs an \(\mathcal{O}(N)\) computational overhead per step.

This paper introduces an intrinsic approach based on a fundamental premise: Shouldn't the model already know which parts of the context are relevant?

To address this, the authors propose Declarative Attention (DA), a protocol that enables language models to declare where they need to attend directly within their chain-of-thought.

⚙️ How Declarative Attention (DA) Works

DA partitions text generation into three distinct operational modes: 1. <global>: Accesses the full context. 2. <focus>: Targets a specific region of interest. 3. <local>: Restricts attention to recent output only.

The inference engine parses these declarations similarly to tool calls, allowing it to bypass and skip reading the vast majority of the KV cache.

⚙️ How Declarative Attention (DA) Works

DA partitions text generation into three distinct operational modes: 1. <global>: Accesses the full context. 2. <focus>: Targets a specific region of interest. 3. <local>: Restricts attention to recent output only.

The inference engine parses these declarations similarly to tool calls, allowing it to bypass and skip reading the vast majority of the KV cache.

📊 Key Results & Findings

  • Zero-Shot Evaluation: Evaluated across 15 long-context tasks using off-the-shelf models (Gemma-4-31B and Qwen-3.6-27B).
  • Efficiency Gains: Significantly reduced the total number of attended tokens during decoding (52.0% reduction for Gemma; 31.1% for Qwen).
  • Trade-off: Achieved this efficiency with modest accuracy drops (1.27 pp and 2.75 pp, respectively), which decrease further as model scale increases.
  • Future Potential: DA opens up a new axis for sparse attention, paving the way for even higher efficiency through future training-based methodologies.

📊 Key Results & Findings

  • Zero-Shot Evaluation: Evaluated across 15 long-context tasks using off-the-shelf models (Gemma-4-31B and Qwen-3.6-27B).
  • Efficiency Gains: Significantly reduced the total number of attended tokens during decoding (52.0% reduction for Gemma; 31.1% for Qwen).
  • Trade-off: Achieved this efficiency with modest accuracy drops (1.27 pp and 2.75 pp, respectively), which decrease further as model scale increases.
  • Future Potential: DA opens up a new axis for sparse attention, paving the way for even higher efficiency through future training-based methodologies.