跳转至

超越结果奖励：面向深度搜索智能体的步级自蒸馏策略优化

文章背景与核心概要

深度搜索智能体（Deep Search Agents）在强化学习训练中常常面临性能瓶颈，因为它们的运行轨迹通常跨越数十个步骤，而标准的强化学习每条轨迹仅提供单一的结果奖励，导致信用分配（Credit Assignment）过于稀疏。

尽管在线自蒸馏（On-Policy Self-Distillation, OPSD）利用模型自身的 Logits 作为密集的词元级教师信号，但将其直接应用於搜索智能体时会暴露出一个根本性矛盾：教师模型拥有特权信息（例如正确答案），其生成的分布与学生模型基于探索的推理过程存在系统性差异。简单的蒸馏会迫使学生继承这种信息不对称，而不是去学习更好的搜索策略。

本文通过两大核心贡献解决了这一矛盾：1. 证据锚点（Evidence Anchors）：构建从网络中提取的简洁、步级证据片段作为特权信息，在不暴露完整答案路径的前提下捕捉关键推理步骤。2. 步级自蒸馏策略优化（SSPO）：该方法将教师与学生的意见分歧转化为 GRPO 中的步级优势权重，且仅应用于错误的轨迹。通过解耦“更新什么”与“更新幅度”，结果奖励决定策略改变的方向，而教师模型则在每一步调节其更新幅度；正确的轨迹则保持原样以保留其多样性。在 Qwen3-8B 上的评估表明，SSPO 在 BrowseComp、GAIA 和 FRAMES 等基准测试中均持续优于 GRPO，其性能超越或持平于进行了双倍梯度步数训练的 GRPO，同时每步仅增加约 5% 的单次额外前向传播开销。

  • Authors: Haoze Wu, Chuqiao Kuang, Tianyi Zhuang, Xiaoguang Li
  • Submitted: August 13, 2026
  • Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
  • arXiv ID: 2608.12764
  • Authors: Haoze Wu, Chuqiao Kuang, Tianyi Zhuang, Xiaoguang Li
  • Submitted: August 13, 2026
  • Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
  • arXiv ID: 2608.12764

Abstract Summary

Deep search agents often struggle with reinforcement learning because they operate over trajectories spanning dozens of steps, while standard RL provides only a single outcome reward per trajectory—offering credit assignment that is far too sparse.

Abstract Summary

Deep search agents often struggle with reinforcement learning because they operate over trajectories spanning dozens of steps, while standard RL provides only a single outcome reward per trajectory—offering credit assignment that is far too sparse.

While On-Policy Self-Distillation (OPSD) uses a model's own logits as dense token-level teachers, applying it to search agents reveals a fundamental tension: the teacher has access to privileged information (such as the correct answer) and produces a distribution systematically different from the student's exploration-based reasoning. Naive distillation forces the student to inherit this information asymmetry instead of learning better search strategies.

While On-Policy Self-Distillation (OPSD) uses a model's own logits as dense token-level teachers, applying it to search agents reveals a fundamental tension: the teacher has access to privileged information (such as the correct answer) and produces a distribution systematically different from the student's exploration-based reasoning. Naive distillation forces the student to inherit this information asymmetry instead of learning better search strategies.

This paper resolves the tension through two primary contributions: 1. Evidence Anchors: Concise, step-level evidence snippets extracted from the web are constructed as privileged information, capturing key reasoning steps without exposing the entire answer path. 2. Step-Level Self-Distilled Policy Optimization (SSPO): This approach converts teacher-student disagreement into step-level advantage weights within GRPO, applied exclusively to incorrect trajectories.

This paper resolves the tension through two primary contributions: 1. Evidence Anchors: Concise, step-level evidence snippets extracted from the web are constructed as privileged information, capturing key reasoning steps without exposing the entire answer path. 2. Step-Level Self-Distilled Policy Optimization (SSPO): This approach converts teacher-student disagreement into step-level advantage weights within GRPO, applied exclusively to incorrect trajectories.

By decoupling what to update from how much to update, the outcome reward determines the direction of policy change, while the teacher modulates its magnitude at each step. Correct trajectories remain untouched to preserve their diversity.

By decoupling what to update from how much to update, the outcome reward determines the direction of policy change, while the teacher modulates its magnitude at each step. Correct trajectories remain untouched to preserve their diversity.

Results

Evaluated on Qwen3-8B, SSPO consistently outperforms GRPO across benchmarks including BrowseComp, GAIA, and FRAMES. It surpasses or matches GRPO trained with twice as many gradient steps while adding only about a 5% overhead per step from a single additional forward pass.

Results

Evaluated on Qwen3-8B, SSPO consistently outperforms GRPO across benchmarks including BrowseComp, GAIA, and FRAMES. It surpasses or matches GRPO trained with twice as many gradient steps while adding only about a 5% overhead per step from a single additional forward pass.