Search-G1：基于表征内在奖励的扎根搜索智能体

文章背景与核心概要

检索增强型语言智能体在处理复杂任务时，面临着何时进行检索以及如何有效利用检索证据的核心挑战。传统的奖励机制往往依赖稀疏的结果监督，难以区分真正的证据扎根与冗余搜索，而依赖人类标注或大模型裁判（LLM-as-a-judge）的方法则成本高昂。另一方面，基于策略内部信号（如熵或Token似然）的方法虽然廉价，但主要反映模型置信度而非对证据的实际依赖。

为了解决这一难题，本文推出了 Search-G1 框架。该框架引入了两个经过干预校准的读出机制（readouts）：一个是预测闭卷充分性的提示状态读出，用于定义策略相关的“检索必要性”；另一个是通过评估答案阶段对证据删除的敏感度来估计“证据依赖性”的答案提交读出。两者结合，在无需人工过程标注或训练期大模型推理的情况下，优化了扎根度与搜索成本之间的权衡。

随着强化学习不断改变策略表征，Search-G1 能够周期性地在最新检查点的轨迹上重新拟合这两个读出器，从而实现奖励与策略的共同演化。实验表明，Search-G1 在保持具有竞争力的任务准确率的同时，显著缩短了响应端的轨迹长度，提升了整体效率。

执行摘要 (Executive Summary)

Search-G1 是一个新颖的框架，旨在通过利用基于表征的内在奖励（representation-based intrinsic rewards）来改进检索增强型语言智能体。传统的检索增强语言模型奖励机制要么依赖稀疏的结果监督（无法区分真正的扎根检索与冗余搜索），要么需要昂贵的标注和大模型裁判。相反，内部策略侧的信号（如熵或Token似然）主要捕获模型置信度，而不是实际的证据依赖性。

Search-G1 引入了两个经过干预校准的读出机制： 1. 提示状态读出（Prompt-state readout）：预测闭卷充分性，定义了策略相关的检索必要性。 2. 答案提交读出（Answer-commit readout）：通过答案阶段对证据删除的敏感度来估计证据依赖性。

这些机制共同优化了扎根与搜索成本之间的权衡，在策略优化期间既不需要人工过程标注，也不需要大模型推理。

Search-G1 is a novel framework designed to improve search-augmented language agents by utilizing representation-based intrinsic rewards. Traditional reward mechanisms for retrieval-augmented language models either rely on sparse outcome supervision (failing to distinguish genuine grounding from redundant searches) or require costly annotations and LLM judges. Conversely, internal policy-side signals (like entropy or token likelihood) primarily capture model confidence rather than actual evidence reliance.

Search-G1 introduces two intervention-calibrated readouts: 1. Prompt-state readout: Predicts closed-book sufficiency, defining policy-relative retrieval necessity. 2. Answer-commit readout: Estimates evidence reliance via answer-stage sensitivity to evidence deletion.

Together, these mechanisms optimize the grounding-to-search-cost trade-off without requiring human process annotations or LLM inference during policy optimization.

论文元数据 (Paper Metadata)

arXiv ID: arXiv:2608.07531 [cs.CL]
研究主题: 计算与语言 (cs.CL); 人工智能 (cs.AI)
作者: Ruoxi Cheng, Haoxuan Ma, Hongyi Zhang, Junming Zhang, Ranjie Duan, Qiaolin Xia, Hao Wang, Yu Lu, Haibo Shi, Xingjun Ma
提交/修订日期: 2026年7月24日提交；2026年8月18日最后修订 (v2)
官方代码库: GitHub - Rosy0912/Search-G1

arXiv ID: arXiv:2608.07531 [cs.CL]

Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)

Authors: Ruoxi Cheng, Haoxuan Ma, Hongyi Zhang, Junming Zhang, Ranjie Duan, Qiaolin Xia, Hao Wang, Yu Lu, Haibo Shi, Xingjun Ma

Submitted / Revised: Submitted on 24 Jul 2026; Last revised 18 Aug 2026 (v2)

Official Code Repository: GitHub - Rosy0912/Search-G1

摘要 (Abstract)

搜索增强型语言智能体应该只在必要时检索外部信息，并将其答案建立在检索到的证据之上。现有的外部奖励提供稀疏的结果监督，或者通过过程标注和LLM裁判提供更丰富的反馈。结果奖励易于扩展，但无法区分扎根检索与冗余搜索，而更丰富的信号在训练过程中需要昂贵的标注或推理。

基于策略侧信号（如熵、似然或信息增益）的内部奖励虽然分级且评估成本低廉，但主要反映模型置信度而非证据扎根。我们提出了 Search-G1，这是一个基于表征的内在奖励框架，通过两个经过干预校准的读出机制来测量智能体答案的操作扎根性： * 提示状态读出预测闭卷充分性，其补集定义了策略相关的检索必要性。 * 答案提交读出通过答案阶段对证据删除的敏感度来估计证据依赖性。

它们共同作用：当估计检索必要且答案对证据敏感时，对正确的搜索轨迹给予额外奖励；当闭卷知识足够时，偏好正确的直接答案；并对重复搜索进行惩罚。校准后，奖励评分在策略优化期间既不需要过程标注，也不需要大模型即裁判（LLM-as-judge）的推理。

由于强化学习会改变策略表征，Search-G1 会定期在最新检查点的轨迹上重新拟合这两个读出器，使奖励与策略共同演化。在多个基于搜索的问答基准和两个模型尺度上的实验表明，Search-G1 改善了扎根与搜索成本的权衡，在具有竞争力的任务准确率下生成了更短的响应侧轨迹。

Search-augmented language agents should retrieve external information only when necessary and ground their answers in retrieved evidence. Existing external rewards provide either sparse outcome supervision or richer feedback from process annotations and LLM judges. Outcome rewards scale readily but cannot distinguish grounded retrieval from redundant search, whereas richer signals require costly annotation or inference during training.

Internal rewards based on policy-side signals such as entropy, likelihood, or information gain are graded and inexpensive to evaluate, yet mainly reflect model confidence rather than evidence grounding. We propose Search-G1, a representation-based intrinsic reward framework that measures the operational grounding of an agent's answers through two intervention-calibrated readouts: * A prompt-state readout predicts closed-book sufficiency, whose complement defines policy-relative retrieval necessity. * An answer-commit readout estimates evidence reliance from answer-stage sensitivity to evidence deletion.

Together, they provide additional credit to correct searched trajectories when retrieval is estimated necessary and the answer is evidence-sensitive, favor correct direct answers when closed-book knowledge suffices, and penalize repeated search. After calibration, reward scoring requires neither process annotations nor LLM-as-judge inference during policy optimization.

Because reinforcement learning changes policy representations, Search-G1 periodically refits both readouts on trajectories from the latest checkpoint, allowing the reward to co-evolve with the policy. Experiments across multiple search-based question-answering benchmarks and two model scales show that Search-G1 improves the grounding–search-cost trade-off, producing shorter response-side trajectories at competitive task accuracy.

提交历史 (Submission History)

[v1] 2026年7月24日，星期五，15:22:57 UTC
[v2] 2026年8月18日，星期二，05:20:56 UTC （当前版本）

[v1] Fri, 24 Jul 2026, 15:22:57 UTC

[v2] Tue, 18 Aug 2026, 05:20:56 UTC (current version)