跳转至

最佳实践评判器优化 (BPCO)

文章背景与核心概要

在大语言模型的强化学习训练中，以 GRPO 为代表的基于群组的方法广受欢迎，因为它们通过对每个提示词采样多个响应，从而绕过了对独立评判器（critic）模型的需求。然而，理论上如果有一个可靠的评判器，仅需单个响应就能估计出 Token 级别的优势，但标准的评判器训练方法往往存在不稳定性。

本文引入了最佳实践评判器优化 (Best Practice Critic Optimization, BPCO)，这是一种旨在解决这些训练不稳定性的稳健方案。BPCO 结合了 DPPO、值域受限的价值预测、蒙特卡洛价值目标、未归一化的策略优势以及长度自适应的广义优势估计等关键组件。由于评判器仅在训练阶段使用，BPCO 还允许其条件化于对策略隐藏的、决定奖励的信息（如参考答案或评分标准）。在各项数学推理基准测试中（模型规模从 1.5B 参数到 30B-A3B 混合专家模型不等），BPCO 持续超越了标准的评判器基线，并在每个提示词仅采样一个响应的情况下，达到了或超过了基于群组的基线水平。

摘要 (Abstract)

Group-based reinforcement learning methods such as GRPO for large language models avoid training a critic by sampling multiple responses for each prompt. A reliable critic could instead estimate token-level advantages from one response, but standard critic-based training recipes are often unstable. We study this instability and develop Best Practice Critic Optimization (BPCO), a recipe that combines DPPO, value predictions bounded to the reward range, Monte Carlo value targets, unnormalized policy advantages, and length-adaptive generalized advantage estimation. Because the critic is used only during training, BPCO can also condition it on reward-defining information, such as a reference answer or grading rubric, that is hidden from the policy. Controlled experiments isolate the effect of each design choice. Across mathematical reasoning tasks with models ranging from 1.5B parameters to 30B-A3B mixtures of experts, BPCO improves a strong critic-based baseline consistently, and matches or exceeds a group-based baseline while sampling one response per prompt. The same recipe also improves learning with rubric-based rewards. These results show that a carefully designed critic provides a reliable alternative to group-relative advantage estimation.

针对大语言模型，基于群组的强化学习方法（如 GRPO）通过为每个提示词采样多个响应，避免了训练评判器（critic）。相反，一个可靠的评判器本可以通过单个响应估计 Token 级别的优势，但标准的基于评判器的训练方案往往不稳定。我们研究了这种不稳定性，并开发了最佳实践评判器优化 (Best Practice Critic Optimization, BPCO) 方案，该方案结合了 DPPO、限定在奖励范围内的价值预测、蒙特卡洛价值目标、未归一化的策略优势以及长度自适应的广义优势估计。由于评判器仅在训练期间使用，BPCO 还可以将其条件化于对策略隐藏的、决定奖励的信息（如参考答案或评分标准）。受控实验孤立了每个设计选择的效果。在涵盖从 1.5B 参数到 30B-A3B 混合专家模型的各项数学推理任务中，BPCO 持续改进了一个强基于评判器的基线，并且在每个提示词采样一个响应的同时，匹配或超过了基于群组的基线。同一方案还改善了基于评分标准奖励的学习。这些结果表明，精心设计的评判器为群组相对优势估计提供了一种可靠的替代方案。

最佳实践评判器优化 (Best Practice Critic Optimization)

总结 (Summary)

Group-based reinforcement learning methods (like GRPO) are popular for training large language models because they bypass the need for a separate critic model by sampling multiple responses per prompt. While a reliable critic could theoretically estimate token-level advantages from a single response, standard critic-based training methods frequently suffer from instability.

以 GRPO 为代表的基于群组的强化学习方法在训练大语言模型时非常流行，因为它们通过对每个提示词采样多个响应，绕过了对独立评判器模型的需求。尽管理论上可靠的评判器可以通过单个响应估计 Token 级别的优势，但标准的评判器训练方法经常遭遇不稳定性的困扰。

This paper introduces Best Practice Critic Optimization (BPCO), a robust recipe designed to resolution these training instabilities. BPCO combines several key components: * DPPO * Value predictions bounded within the reward range * Monte Carlo value targets * Unnormalized policy advantages * Length-adaptive generalized advantage estimation

本文引入了最佳实践评判器优化 (Best Practice Critic Optimization, BPCO)，这是一种旨在解决这些训练不稳定性的稳健方案。BPCO 结合了几个关键组件： * DPPO * 范围受限于奖励区间内的价值预测 * 蒙特卡洛价值目标 * 未归一化的策略优势 * 长度自适应的广义优势估计

Because the critic is exclusively used during training, BPCO can condition it on hidden, reward-defining information (such as reference answers or grading rubrics) that remain concealed from the policy. Across various mathematical reasoning benchmarks (ranging from 1.5B parameters to 30B-A3B mixture-of-experts models), BPCO consistently improves upon standard critic-based baselines and matches or exceeds group-based baselines while requiring only a single response per prompt.

由于评判器仅在训练期间使用，BPCO 可以将其条件化于对策略隐藏的、决定奖励的信息（例如参考答案或评分标准）。在各种数学推理基准测试中（从 1.5B 参数到 30B-A3B 混合专家模型不等），BPCO 持续改进了标准的基于评判器的基线，并且在每个提示词仅需采样一个响应的前提下，匹配或超过了基于群组的基线。

论文元数据 (Paper Metadata)

  • arXiv ID: arXiv:2608.23566
  • Primary Subject: Machine Learning (cs.LG)
  • Secondary Subjects: Artificial Intelligence (cs.AI), Computation and Language (cs.CL)
  • Authors: Penghui Qi, Xiangxin Zhou, Wee Sun Lee
  • Submission Dates:
  • Submitted on 24 Aug 2026 (v1)
  • Last revised 25 Aug 2026 (v2)
  • DOI: 10.48550/arXiv.2608.23566
  • Official Code Repository: GitHub - QPHutu/golden_critic