跳转至

RTPO：用于稳定智能体强化学习训练的反向轮次策略优化

文章背景与核心概要

随着大语言模型（LLM）在复杂推理、外部工具调用及迭代搜索任务中的应用日益广泛，利用强化学习（RL）训练多轮智能体工作流已成为提升模型性能的关键路径。然而，该训练过程极不稳定，随着交互轮次的增加，模型性能往往会出现严重退化。

本文深入分析了导致不稳定的三个核心因素：回放与训练的上下文不匹配、稀疏奖励下的轮次级信用分配不足，以及因优化不同长度轨迹而产生的异步策略漂移。针对这些源于扁平化轨迹优化结构的根本问题，作者提出了“反向轮次策略优化”（RTPO）。该方法通过将多轮回放组织为稀疏反向树，并采用时间反向顺序进行策略更新，有效解决了上下文不匹配与异步漂移问题，在多项基准测试中显著优于现有基线模型。

📌 摘要

训练具有强化学习（RL）的多轮智能体工作流，使大语言模型（LLM）能够执行复杂的推理、利用外部工具并执行迭代搜索。然而，这一过程仍然极其不稳定，随着交互轮次数量的增加，往往会导致严重的性能下降。

Training multi-turn agentic workflows with reinforcement learning (RL) enables large language models (LLMs) to perform complex reasoning, leverage external tools, and execute iterative searches. However, this process remains notoriously unstable, often leading to severe performance degradation as the number of interaction turns increases.

本文确定了三个紧密耦合的不稳定来源： 1. 回放-训练上下文不匹配 2. 稀疏终端奖励下的弱轮次级信用分配 3. 因在不同策略版本下优化短轨迹和长轨迹而导致的异步策略漂移

This paper identifies three tightly coupled sources of instability: 1. Rollout-training context mismatch 2. Weak turn-level credit assignment under sparse terminal rewards 3. Asynchronous policy drift caused by optimizing short and long trajectories under different policy versions

为了解决这些源于扁平化轨迹优化中共同结构起源的问题，作者引入了反向轮次策略优化（RTPO）。

To resolve these issues—which stem from a common structural origin in flattened trajectory optimization—the authors introduce Reverse-Turn Policy Optimization (RTPO).

🛠️ 关键贡献与方法论

  • 统一的反向轮次公式： RTPO 将多轮回放组织为稀疏反向树，通过结构化更新，系统地将决策与后续的延续动作对齐。
  • 时间反向顺序更新： 通过按时间反向顺序执行轮次级策略更新，该框架实现了因果一致的信用分配，并利用策略内延续来主动控制异步漂移。
  • 理论保证： 本文提供了数学证明，表明 RTPO 成功消除了上下文不匹配和异步漂移，减轻了信用偏差，并收敛于递归最优性。
  • Unified Reverse-Turn Formulation: RTPO organizes multi-turn rollouts as sparse reverse trees, structuring updates to systematically align decisions with downstream continuations.
  • Temporal Reverse-Order Updates: By performing turn-level policy updates in temporal reverse order, the framework achieves causally consistent credit assignment and on-policy continuation to actively control asynchronous drift.
  • Theoretical Guarantees: The paper provides mathematical proofs showing that RTPO successfully eliminates context mismatch and asynchronous drift, mitigates credit bias, and converges to recursive optimality.

📊 实验结果

在已建立的多轮智能体 RL 基准测试中，RTPO 显著优于竞争基线： * 较轨迹级基线提升 +21.50%。 * 较轮次级基线提升 +10.76%。

这些结果突显了 RTPO 在为先进工具使用 AI 智能体提供稳定、可扩展的训练基础方面的有效性。

Evaluated across established multi-turn agentic RL benchmarks, RTPO outperforms competing baselines significantly: * +21.50% improvement over trajectory-level baselines. * +10.76% improvement over turn-level baselines.

These results highlight RTPO's efficacy in providing a stable, scalable training foundation for advanced tool-using AI agents.

🔗 全文与资源

[内容根据 CC BY 4.0 协议通过 arXiv 授权]
(关联图像资源来自存储库引用：license icon)

[Content licensed under CC BY 4.0 via arXiv]
(Associated image asset from repository reference preserved: license icon)