跳转至

对比分支策略优化 (CBPO)

文章背景与核心概要

对比分支策略优化（Contrastive Branch Policy Optimization, CBPO）是一种旨在提升语言模型中可验证奖励强化学习（RLVR）效果的新型框架。尽管 RLVR 允许模型与外部工具交互，但它常受限于稀疏的结果奖励，难以识别出具体哪些中间决策导致了最终成功。

CBPO 通过将预算分配与 Token 级别的信用分配解耦来解决这一问题。它利用生成熵以及路径/节点级别的衰减，有效地在不同轨迹间分配资源。此外，该研究引入了“对比分支价值”（Contrastive Branch Value, CBV）——这是一种基于结果的度量指标，用于估计“精确前缀组”内的局部决策敏感度，从而在无需过程级标注的情况下优化信用分配。

对比分支策略优化 (Contrastive Branch Policy Optimization)

作者： Ying Wang, Changlin Qiu, Bang Lin, Linbo Jin, Wen Jiang, Zhe Sun, Jingli Yang
日期： 2026年8月25日
学科： 机器学习 (cs.LG); 人工智能 (cs.AI)
arXiv ID: 2608.24300

摘要

Contrastive Branch Policy Optimization (CBPO) is a novel framework designed to improve Reinforcement Learning with Verifiable Rewards (RLVR) in language models. While RLVR allows models to interact with external tools, it often suffers from sparse outcome rewards that fail to identify which specific intermediate decisions lead to success. CBPO addresses this by decoupling budget allocation from token-level credit assignment. By utilizing generation entropy and path/node-level decay, it effectively distributes resources across trajectories. Furthermore, it introduces the Contrastive Branch Value (CBV)—an outcome-based metric that estimates local decision sensitivity within "exact-prefix groups"—to refine credit assignment without requiring process-level annotations.

对比分支策略优化 (CBPO) 是一种旨在改进语言模型中可验证奖励强化学习 (RLVR) 的新型框架。虽然 RLVR 允许模型与外部工具进行交互，但它通常会受到稀疏结果奖励的困扰，无法识别出哪些具体的中间决策导致了成功。CBPO 通过将预算分配与 Token 级别的信用分配解耦来解决这一问题。通过利用生成熵和路径/节点级别的衰减，它有效地在轨迹间分配资源。此外，它引入了“对比分支价值”(CBV)——这是一种基于结果的度量标准，用于估计“精确前缀组”内的局部决策敏感度，从而在无需过程级标注的情况下优化信用分配。

关键特性与方法论 (Key Features & Methodology)

1. 解耦优化 (Disentangled Optimization)

CBPO separates the two primary challenges of branch sampling: * Budget Allocation: Uses generation entropy to screen candidate branch positions across the entire response. * Credit Assignment: Employs path-level and node-level decay to prevent exploration from collapsing onto a limited set of paths or adjacent tokens.

CBPO 将分支采样的两个主要挑战分离开来： * 预算分配： 使用生成熵来筛选整个响应中的候选分支位置。 * 信用分配： 采用路径级和节点级的衰减，以防止探索过程坍缩到有限的路径集或相邻的 Token 上。

2. 对比分支价值 (Contrastive Branch Value, CBV)

The method identifies "exact-prefix groups"—parent trajectories and their branches that share an identical token prefix. The reward variation within these groups defines the CBV, providing a robust estimate of local decision sensitivity that rescales continuation advantages.

该方法识别出“精确前缀组”——即共享相同 Token 前缀的父轨迹及其分支。这些组内的奖励变化定义了 CBV，提供了一种稳健的局部决策敏感度估计，用于重新缩放延续优势。

3. 高效信用分割 (Efficient Credit Segmentation)

To avoid the issue of duplicated gradients on shared tokens, CBPO partitions trajectories into non-overlapping credit segments when multiple nodes are selected, ensuring more precise learning signals.

为了避免共享 Token 上重复梯度的问题，当选择多个节点时，CBPO 将轨迹划分为互不重叠的信用片段，从而确保了更精确的学习信号。

性能亮点 (Performance Highlights)

CBPO has been validated across ten benchmarks, covering both mathematical reasoning and knowledge-intensive search. Key findings include: * Superior Accuracy: Consistently outperforms state-of-the-art policy optimization and branch-based methods. * Scalability: Achieves the highest macro-average accuracy across two different model scales. * Practicality: Requires only outcome rewards, eliminating the need for expensive process-level human or model annotations.

CBPO 已在十个基准测试中得到验证，涵盖了数学推理和知识密集型搜索。主要发现包括： * 卓越的准确性： 在性能上持续优于最先进的策略优化和基于分支的方法。 * 可扩展性： 在两种不同的模型规模下均实现了最高的宏观平均准确率。 * 实用性： 仅需结果奖励，无需昂贵的过程级人工或模型标注。

元数据与访问 (Metadata & Access)