跳转至

文章背景与核心概要

结果监督（Outcome-supervised）的搜索智能体在面对复杂任务时往往表现不佳，因为最终的奖励无法在中间错误累积之前对其进行定位。与此同时，依赖静态的评判器（critic）也无法奏效，因为随着智能体自身能力的提升，其失败模式也会随之改变。

本文介绍了 CAFE（Coupled Agent–Feedback Evolution，耦合智能体与反馈进化） 框架。在该框架中，共享参数的模型在“搜索智能体”和“评判器”的角色之间交替转换。通过将在线强化学习（RL）与比较反馈估计、优势塑造（advantage shaping）以及离线偏好优化相结合，CAFE 实现了智能体与其反馈机制的共同进化。在七个智能体搜索基准上的实验表明，CAFE 的性能优于现有的基于 RL 的搜索智能体，能够有效地泛化到领域外任务，并显著减少答案层面的幻觉。

CAFE: Self-Improving Search Agents Need Co-Evolving Feedback

arXiv: 2608.24794 [cs.AI]
Submitted: August 25, 2026
Authors: Boyang Liu, Senjie Jin, Peixin Wang, Zhangyue Yin, Yibo Wang, Yuhao Zhou, Xinbing Liang, Shizheng Zhu, Yuhui Wang, Jingqi Tong, Zhiheng Xi, Jiazheng Zhang, Clive Bai, Clarenceai, Blaze Chen, Tao Gui, Qi Zhang, and Xuanjing Huang

📌 Executive Summary

Outcome-supervised search agents often struggle because terminal rewards fail to localize intermediate errors before they compound. Meanwhile, relying on a static critic falls short because failure patterns shift as the agent improves.

This paper introduces CAFE (Coupled Agent–Feedback Evolution), a framework where a shared-parameter model alternates between acting as a search agent and a critic. By combining online reinforcement learning (RL) with comparative feedback estimates, advantage shaping, and offline preference optimization, CAFE enables the agent and its feedback mechanism to co-evolve. Experiments across seven agentic search benchmarks demonstrate that CAFE outperforms existing RL-based search agents, generalizes effectively to out-of-domain tasks, and significantly reduces answer-level hallucinations.

结果监督的搜索智能体在处理任务时往往举步维艰，因为终端奖励无法在中间错误累积之前对其进行定位。与此同时，依赖静态的评判器也不敷使用，因为随着智能体的改善，失败模式会发生转移。

本文引入了 CAFE（耦合智能体与反馈进化，Coupled Agent–Feedback Evolution） 框架，在该框架中，共享参数的模型在搜索智能体和评判器的角色之间交替转换。通过将在线强化学习（RL）与比较反馈估计、优势塑造和离线偏好优化相结合，CAFE 使智能体及其反馈机制能够共同进化。在七个智能体搜索基准上的实验表明，CAFE 的表现优于现有的基于 RL 的搜索智能体，能够有效地泛化到域外任务，并显着减少答案级别的幻觉。

📄 Abstract

Outcome-supervised search agents learn when and how to retrieve evidence, but terminal rewards neither localize intermediate errors nor redirect an ongoing trajectory before those errors compound. Treating corrective feedback as a learned in-trajectory intervention couples the two roles: the agent must decide when to request and use feedback, while the critic must infer useful corrections from outcome-confounded rollouts whose failure patterns shift as the agent improves.

We introduce CAFE (Coupled Agent--Feedback Evolution), a framework in which a shared-parameter model alternates between search-agent and critic roles. CAFE initializes feedback-conditioned recovery from trajectories built around the base agent's own failures, then couples online and offline optimization. During online RL, a comparative feedback estimate uses a prompt-level call--skip success gap to shape request returns, while feedback-aware advantage shaping reweights token advantages before and after feedback. Offline, rollout-derived preference optimization learns feedback from matched successful and unsuccessful trajectories.

On seven agentic search benchmarks, CAFE outperforms the evaluated RL-based search agents on average, retains its gains across all six out-of-domain benchmarks, and reduces answer-level hallucinations. One-sided ablations show that improving only the agent or only the critic eventually plateaus, whereas alternating the two updates continues to improve performance. These findings suggest that a self-improving search agent needs feedback that co-evolves with the policy it guides.

结果监督的搜索智能体学习何时以及如何检索证据，但终端奖励既不能定位中间错误，也不能在这些错误累积之前重定向正在进行的轨迹。将纠正性反馈视为一种学习到的轨迹内干预，将这两个角色耦合在了一起：智能体必须决定何时请求和使用反馈，而评判器必须从结果混杂的展开（rollouts）中推断出有用的纠正，而这些失败模式会随着智能体的改进而发生变化。

我们引入了 CAFE（耦合智能体与反馈进化，Coupled Agent--Feedback Evolution） 框架，在该框架中，共享参数的模型在搜索智能体和评判器角色之间交替。CAFE 从围绕基础智能体自身失败构建的轨迹中初始化反馈条件恢复，然后将在线和离线优化结合起来。在在线 RL 期间，比较反馈估计使用提示级别的调用-跳过成功差距来塑造请求回报，而反馈感知的优势塑造则在反馈前后重新加权标记优势（token advantages）。在离线状态下，由展开推导出的偏好优化从匹配的成功和不成功轨迹中学习反馈。

在七个智能体搜索基准测试中，CAFE 平均优于所评估的基于 RL 的搜索智能体，在所有六个域外基准测试中均保留了其收益，并减少了答案级别的幻觉。单边消融实验表明，仅改进智能体或仅改进评判器最终会趋于平稳，而交替进行这两项更新则会持续提升性能。这些发现表明，自我改进的搜索智能体需要与它所引导的策略共同进化的反馈。