跳转至

Co-RL：多智能体强化学习中多元群组涌现的无监督推理

文章背景与核心概要

强化学习（RL）已被证明是提升语言和视觉语言模型推理能力的关键技术，但传统方法严重依赖昂贵的真实真值监督（可验证奖励），而随着任务复杂度的提升，这类监督信号变得愈发稀缺。自我奖励型强化学习虽然通过让模型评估自身输出来消除外部注释的需求，但也常常面临训练崩溃、自我强化偏差以及响应多样性降低等问题。

为了克服这些局限性，本文引入了一种新颖的多智能体强化学习框架 —— Co-RL。Co-RL 利用由解耦模型组成的多元群组（不共享参数），通过同伴派生的奖励信号相互监督和优化。通过有意增加群组多样性（包括异构模型系列、不同的模型规模以及改写的训练样本），该框架有效地打破了相关的错误循环。在不依赖任何真实真值标签的情况下，Co-RL 持续提升了各个基准测试中的推理性能，其表现达到甚至超越了监督同类方法： * 纯文本大语言模型基准： 在七个基准测试中平均性能提升 3.0% – 8.6%。 * 多模态视觉语言模型基准： 在四个基准测试中平均性能提升 2.3% – 7.2%。

Summary

Reinforcement learning (RL) significantly enhances reasoning in language and vision-language models, but traditional approaches heavily rely on costly ground-truth supervision (verifiable rewards) that become scarce as task complexity scales. While self-rewarding RL eliminates the need for external annotations by having models evaluate their own outputs, it often suffers from training collapse, self-reinforcing biases, and reduced response diversity.

To overcome these limitations, this paper introduces Co-RL, a novel multi-agent reinforcement learning framework. Co-RL leverages a diverse cohort of decoupled models (sharing no parameters) that supervise and optimize one another through peer-derived reward signals. By intentionally increasing cohort diversity—incorporating heterogeneous model families, varying sizes, and rephrased training samples—the framework effectively breaks correlated error loops. Without relying on any ground-truth labels, Co-RL consistently improves reasoning performance across benchmarks, matching or outperforming supervised counterparts: * Text-only LLM Benchmarks: Average performance gains of 3.0% – 8.6% across seven benchmarks. * Multimodal VLM Benchmarks: Average performance gains of 2.3% – 7.2% across four benchmarks.

强化学习（RL）显著增强了语言和视觉语言模型的推理能力，但传统方法严重依赖成本高昂的真实真值监督（可验证奖励），随着任务复杂度的提升，这种监督信号变得日益稀缺。自我奖励强化学习通过让模型评估自身的输出来消除对外部注释的需求，但它往往遭受训练崩溃、自我强化偏差以及响应多样性降低的困扰。

为了克服这些局限性，本文介绍了 Co-RL，这是一种新颖的多智能体强化学习框架。Co-RL 利用了一个由解耦模型组成的多元群组（不共享参数），这些模型通过同伴推导出的奖励信号相互监督和优化。通过有意增加群组多样性（结合异构模型系列、不同的规模以及改写的训练样本），该框架有效地打破了相关的错误循环。在不依赖任何真实真值标签的情况下，Co-RL 在各大基准测试中持续提升了推理性能，其表现与监督同类方法相当甚至更胜一筹： * 纯文本 LLM 基准： 在七个基准测试中平均性能提升 3.0% – 8.6%。 * 多模态 VLM 基准： 在四个基准测试中平均性能提升 2.3% – 7.2%。

Abstract

Reinforcement learning (RL) has emerged as a powerful approach for improving reasoning in language and vision-language models, yet its strongest successes still depend heavily on ground-truth supervision (e.g., verifiable reward). Such annotations are costly to obtain and become increasingly scarce as reasoning capabilities advance beyond what humans can reliably evaluate. Self-rewarding RL reduces this dependence by enabling models to derive reward signals from their own completions. However, training solely on self-generated feedback can reinforce existing biases and suboptimal behaviors, reduce response diversity, and ultimately lead to homogenized responses and training collapse.

In this work, we show that unsupervised reasoning can emerge through cooperative multi-agent training. We introduce Co-RL, a framework in which multiple decoupled models, sharing no parameters, are simultaneously optimized through RL using rewards derived from their peers. We further show that increasing cohort diversity, through heterogeneous model families, sizes, and rephrased training samples, reduces the correlated errors that drive self-reinforcing feedback loops. This diversity consistently improves reasoning performance, maintains behavioral diversity, and mitigates training collapse. Across text-only and multimodal domains, Co-RL consistently outperforms the base models and prior label-free approaches, while matching or surpassing supervised methods, without access to any ground-truth labels. Concretely, Co-RL yields average gains of 3.0-8.6% across seven text-only benchmarks for LLMs and 2.3-7.2% across four multimodal benchmarks for VLMs.

强化学习（RL）已成为提升语言和视觉语言模型推理能力的一种强大方法，然而其最显著的成功仍然严重依赖于真实真值监督（例如可验证奖励）。此类注释获取成本高昂，并且随着推理能力超越人类可靠评估的极限，它们变得越来越稀缺。自我奖励强化学习通过使模型能够从自身的补全中推导奖励信号，减少了对这种依赖的需要。然而，仅根据自我生成的反馈进行训练会强化现有的偏差和次优行为，降低响应多样性，并最终导致响应同质化和训练崩溃。

在这项工作中，我们表明无监督推理可以通过协作式多智能体训练涌现出来。我们引入了 Co-RL，这是一个框架，其中多个不共享参数的解耦模型通过使用源自其同伴的奖励，通过 RL 同时进行优化。我们进一步表明，通过异构模型系列、规模和改写的训练样本来增加群组多样性，可以减少驱动自我强化反馈循环的相关错误。这种多样性持续改善了推理性能，保持了行为多样性，并缓解了训练崩溃。在纯文本和多模态领域中，在无法访问任何真实真值标签的情况下，Co-RL 始终优于基础模型和先前的无标签方法，同时匹配或超越了监督方法。具体而言，Co-RL 在 LLM 的七个纯文本基准测试中带来了 3.0-8.6% 的平均收益，在 VLM 的四个多模态基准测试中带来了 2.3-7.2% 的平均收益。

Submission History

  • [v1] Tue, 18 Aug 2026 01:16:02 UTC (10,361 KB)
  • [v2] Wed, 19 Aug 2026 05:41:44 UTC (10,361 KB) — This version
  • [v1] 2026年8月18日星期二 01:16:02 UTC (10,361 KB)
  • [v2] 2026年8月19日星期三 05:41:44 UTC (10,361 KB) — 当前版本