跳转至

文章背景与核心概要

使用强化学习（RL）训练多轮智能体工作流，能够使大语言模型（LLMs）执行复杂推理、调用外部工具并进行迭代搜索。然而，这一过程 notoriously 不稳定，随着交互轮数的增加，性能往往会出现严重退化。本文指出了源自扁平化轨迹优化的三个紧密耦合的不稳定性来源： rollout-training上下文不匹配、稀疏终止奖励下的弱轮次信用分配，以及由不同策略版本优化长短轨迹所引起的异步策略漂移。

为了解决这些挑战，作者提出了反向轮次策略优化（RTPO）。RTPO 将多轮 rollout 组织为稀疏反向树，并按时间逆序执行轮次级策略更新。该方法确保了因果一致的信用分配，消除了上下文不匹配和异步漂移，减少了信用偏差，并收敛至递归最优性。实验表明，RTPO 在多轮智能体 RL 基准测试中显著优于标准的轨迹级和轮次级基线方法。

RTPO: Reverse-Turn Policy Optimization for Stabilizing Agentic RL Training

RTPO: Reverse-Turn Policy Optimization for Stabilizing Agentic RL Training

Summary

Summary

Training multi-turn agentic workflows using reinforcement learning (RL) allows large language models (LLMs) to perform complex reasoning, utilize external tools, and engage in iterative searches. However, this process remains notoriously unstable, often causing performance to severely degrade as the number of turns increases.

Training multi-turn agentic workflows using reinforcement learning (RL) allows large language models (LLMs) to perform complex reasoning, utilize external tools, and engage in iterative searches. However, this process remains notoriously unstable, often causing performance to severely degrade as the number of turns increases.

This paper identifies three tightly coupled sources of instability originating from flattened trajectory optimization: 1. Rollout-training context mismatch 2. Weak turn-level credit assignment under sparse terminal rewards 3. Asynchronous policy drift caused by optimizing short and long trajectories under different policy versions

This paper identifies three tightly coupled sources of instability originating from flattened trajectory optimization: 1. Rollout-training context mismatch 2. Weak turn-level credit assignment under sparse terminal rewards 3. Asynchronous policy drift caused by optimizing short and long trajectories under different policy versions

To resolve these challenges, the authors propose Reverse-Turn Policy Optimization (RTPO). RTPO organizes multi-turn rollouts into sparse reverse trees and executes turn-level policy updates in reverse temporal order. This ensures causally consistent credit assignment, eliminates context mismatch and asynchronous drift, reduces credit bias, and converges to recursive optimality. Experiments demonstrate that RTPO outperforms standard trajectory- and turn-level baselines by significant margins.

To resolve these challenges, the authors propose Reverse-Turn Policy Optimization (RTPO). RTPO organizes multi-turn rollouts into sparse reverse trees and executes turn-level policy updates in reverse temporal order. This ensures causally consistent credit assignment, eliminates context mismatch and asynchronous drift, reduces credit bias, and converges to recursive optimality. Experiments demonstrate that RTPO outperforms standard trajectory- and turn-level baselines by significant margins.

Paper Metadata

Paper Metadata

  • arXiv Identifier: arXiv:2608.18682 [cs.AI]
  • Subject: Artificial Intelligence (cs.AI)
  • Authors: Yugu Li, Zehong Cao, Jianglin Qiao, Siyi Hu
  • Submitted On: August 19, 2026
  • Last Revised: August 31, 2026 (Version v3)
  • DOI: 10.48550/arXiv.2608.18682
  • arXiv Identifier: arXiv:2608.18682 [cs.AI]
  • Subject: Artificial Intelligence (cs.AI)
  • Authors: Yugu Li, Zehong Cao, Jianglin Qiao, Siyi Hu
  • Submitted On: August 19, 2026
  • Last Revised: August 31, 2026 (Version v3)
  • DOI: 10.48550/arXiv.2608.18682

Abstract

Abstract

Training multi-turn agentic workflows with reinforcement learning (RL) enables large language models to perform complex reasoning, use external tools, and conduct iterative search beyond single-turn settings. Yet multi-turn RL training remains highly unstable, often causing severe performance degradation as the number of turns increases. Through theoretical analysis, we identify three tightly coupled sources of instability: rollout-training context mismatch, weak turn-level credit assignment under sparse terminal rewards, and asynchronous policy drift when short and long trajectories are optimized under different policy versions. We show that these issues share a common structural origin in flattened trajectory optimization and address them through a unified reverse-turn formulation. We propose Reverse-Turn Policy Optimization (RTPO), which organizes multi-turn rollouts as sparse reverse trees and performs turn-level policy updates in temporal reverse order, aligning each decision with its downstream continuation. RTPO enables causally consistent turn-level credit assignment and on-policy continuation to control asynchronous drift. We provide theoretical guarantees showing that RTPO eliminates context mismatch and asynchronous drift under the proposed turn-level formulation, reduces credit bias, and converges to recursive optimality. Experiments on multi-turn agentic RL benchmarks show that RTPO improves upon trajectory- and turn-level baselines by 21.50% and 10.76%, respectively, highlighting its potential to support more stable training for tool-using agents.

Training multi-turn agentic workflows with reinforcement learning (RL) enables large language models to perform complex reasoning, use external tools, and conduct iterative search beyond single-turn settings. Yet multi-turn RL training remains highly unstable, often causing severe performance degradation as the number of turns increases. Through theoretical analysis, we identify three tightly coupled sources of instability: rollout-training context mismatch, weak turn-level credit assignment under sparse terminal rewards, and asynchronous policy drift when short and long trajectories are optimized under different policy versions. We show that these issues share a common structural origin in flattened trajectory optimization and address them through a unified reverse-turn formulation. We propose Reverse-Turn Policy Optimization (RTPO), which organizes multi-turn rollouts as sparse reverse trees and performs turn-level policy updates in temporal reverse order, aligning each decision with its downstream continuation. RTPO enables causally consistent turn-level credit assignment and on-policy continuation to control asynchronous drift. We provide theoretical guarantees showing that RTPO eliminates context mismatch and asynchronous drift under the proposed turn-level formulation, reduces credit bias, and converges to recursive optimality. Experiments on multi-turn agentic RL benchmarks show that RTPO improves upon trajectory- and turn-level baselines by 21.50% and 10.76%, respectively, highlighting its potential to support more stable training for tool-using agents.

Full-Text & Reference Links

Full-Text & Reference Links