跳转至

文章背景与核心概要

在多智能体大语言模型（LLM）领域中，多个策略通常需要在共享环境中通过协同合作来解决复杂的任务。然而，现有的强化学习框架大多独立优化每个响应或轨迹，即便这些输出共同驱动了单一的状态转移也是如此。这种做法导致优化更新单元与实际系统动作之间产生了脱节。

为了弥补这一差距，本文作者推出了 SRPO（Setwise Relative Policy Optimization，集合相对策略优化）。SRPO 将“活动集”（即单个状态转移所消费的最少输出组合）定义为一个统一的多智能体动作。通过将成员对数比率（log-ratios）组合成基数归一化的集合比率、赋予相对优势（advantage），并应用统一的集合裁剪机制，SRPO 有效实现了任务分工与联合协同进化的统一。

SRPO: Setwise Relative Policy Optimization for Multi-Agent LLMs

Authors: Shengtian Yang, Ziyu Xiong, Yu Li, Yewen Li, Qingpeng Cai, Lei Feng
Subject: Artificial Intelligence (cs.AI)
arXiv ID: arXiv:2609.08452
Submission Date: September 8, 2026

Authors: Shengtian Yang, Ziyu Xiong, Yu Li, Yewen Li, Qingpeng Cai, Lei Feng
Subject: Artificial Intelligence (cs.AI)
arXiv ID: arXiv:2609.08452
Submission Date: September 8, 2026

📌 Summary

Multi-agent Large Language Models (LLMs) solve complex tasks by coordinating multiple policies within a shared environment. However, current reinforcement learning frameworks typically optimize each response or trajectory independently—even when multiple outputs collectively drive a single state transition. This creates a mismatch where the optimization update unit differs from the actual system action.

To bridge this gap, the authors introduce SRPO (Setwise Relative Policy Optimization). SRPO defines the "active set"—the minimal group of outputs consumed by a single transition—as a unified multi-agent action. By combining member log-ratios into a cardinality-normalized set ratio, assigning a relative advantage, and applying a single setwise clip, SRPO effectively unifies task division and joint co-evolution.

📌 Summary

Multi-agent Large Language Models (LLMs) solve complex tasks by coordinating multiple policies within a shared environment. However, current reinforcement learning frameworks typically optimize each response or trajectory independently—even when multiple outputs collectively drive a single state transition. This creates a mismatch where the optimization update unit differs from the actual system action.

To bridge this gap, the authors introduce SRPO (Setwise Relative Policy Optimization). SRPO defines the "active set"—the minimal group of outputs consumed by a single transition—as a unified multi-agent action. By combining member log-ratios into a cardinality-normalized set ratio, assigning a relative advantage, and applying a single setwise clip, SRPO effectively unifies task division and joint co-evolution.

🔍 Key Highlights

  • Unified Formulation: Treats multi-agent coordination and division of labor as actions with varying set sizes.
  • Core Mechanism: Combines individual log-ratios into a cardinality-normalized set ratio with a unified relative advantage and clipping strategy.
  • Robust Performance: Evaluated on mathematical reasoning and multi-turn search tasks across four model scales, supporting fixed, mixed, and dynamically routed workflows with leading macro-average results.
  • Stability Diagnostics: Provides optimization diagnostics to analyze training stability across various event reductions and set sizes.

🔍 Key Highlights

  • Unified Formulation: Treats multi-agent coordination and division of labor as actions with varying set sizes.
  • Core Mechanism: Combines individual log-ratios into a cardinality-normalized set ratio with a unified relative advantage and clipping strategy.
  • Robust Performance: Evaluated on mathematical reasoning and multi-turn search tasks across four model scales, supporting fixed, mixed, and dynamically routed workflows with leading macro-average results.
  • Stability Diagnostics: Provides optimization diagnostics to analyze training stability across various event reductions and set sizes.

(Note: The license icon associated with this article is preserved below to strictly follow formatting requirements)
license icon

(Note: The license icon associated with this article is preserved below to strictly follow formatting requirements)
license icon