跳转至

文章背景与核心概要

在长周期（long-horizon）智能体领域中，强化学习往往在“结果盲区”（outcome-blind）场景下表现不佳，即缺乏真实地面实况成功信号或程序化检查器。虽然多维标准（rubrics）提供了一种替代方案，但传统的评分在整条轨迹中仅提供单一的标量，这对于数十个步骤来说是一个非常薄弱的奖励信号。

为了克服这一挑战，作者提出了 DRACO（Distributing Rubric-based Advantage for Credit Optimization，用于信用优化的基于评分标准的优势分发）。DRACO 在训练过程中动态生成评分标准，以追踪策略不断演进的能力，在每条完成的轨迹上对其进行一次评估，并使用闭式重新分配方法在 GRPO（群组相对策略优化）中分配差异化的每步优势，且无需单独训练归因模块或验证器。该方法在 AppWorld 和 Tau-Bench 任务上展现出了显著的性能提升。

DRACO: Fine-Grained Credit Assignment with Dynamic Rubrics for Long-Horizon Agent Training

DRACO: Fine-Grained Credit Assignment with Dynamic Rubrics for Long-Horizon Agent Training

📋 Summary

📋 Summary

Reinforcement learning for long-horizon agent domains often suffers in "outcome-blind" settings where ground-truth success signals or programmatic checkers are missing. While multi-criteria rubrics offer an alternative, traditional scoring provides only a single scalar per trajectory—a poor reward signal across dozens of steps.

To overcome this, the authors propose DRACO (Distributing Rubric-based Advantage for Credit Optimization). DRACO dynamically generates rubrics during training to track the evolving capabilities of a policy, evaluates them once per completed trajectory, and uses a closed-form redistribution method to assign differentiated, per-step advantages in GRPO without needing a separate trained attribution module or verifiers.

📄 Article Metadata

📄 Article Metadata

  • arXiv ID: arXiv:2609.04094 [cs.AI]
  • Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Software Engineering (cs.SE)
  • Authors: Shubham Gandhi, Saurabh Goyal, Kiran Kate, Yara Rizk
  • Submitted: September 3, 2026
  • DOI: 10.48550/arXiv.2609.04094

🔍 Abstract

具有可验证奖励的强化学习在任务拥有程序化检查器时效果很好，但大多数长周期智能体领域并没有这种检查器。我们在“结果盲区”设定下工作，此时无法获得真实的成功信号。多标准评分是一种提供此类奖励的流行方法；它们每条轨迹评分一次，但在数十个步骤中，单个标量是一个很差的信号。我们提出了 DRACO：用于信用优化的基于评分标准的优势分发。它在训练期间动态生成评分标准以追踪策略不断演进的能力，在每条完成的轨迹上对这些评分标准进行一次评分，并将该判断重新分发到负责已标注评分标准的步骤上，从而在 GRPO 中产生差异化的每步优势。这种重新分配是闭式的，并且不引入任何经过训练的归因模块。在 AppWorld 上，尽管 DRACO 本身没有使用任何验证器，但它比基础模型高出 15.9 个百分点，比使用稀疏地面真实奖励训练的 GRPO 高出 5.3 个百分点。在域外的 Tau-Bench 上，即使没有前沿评判器（frontier judge），它也比基础模型高出 5.3 个百分点，超越了地面真实奖励训练和其他基于评分标准的训练设置。

Reinforcement Learning from Verifiable Rewards works well when a task has a programmatic checker, but most long-horizon agent domains have none. We work in the outcome-blind setting, where ground-truth success signals are not available. Multi-criteria rubrics are a popular way to supply such a reward; they are scored once per trajectory, but a single scalar is a poor signal across tens of steps. We propose DRACO: Distributing Rubric-based Advantage for Credit Optimization. It generates rubrics dynamically during training to track the policy's evolving capability, scores those rubrics once per completed trajectory, and redistributes that judgment over the steps responsible for annotated rubrics to produce differentiated per-step advantages in GRPO. The redistribution is closed-form and does not introduce any trained attribution module. On AppWorld, DRACO gains 15.9 points over the base model and 5.3 points over GRPO trained with a sparse ground-truth reward, despite not using any verifiers itself. On out-of-domain Tau-Bench, it gains 5.3 points over the base model even without a frontier judge, beating both ground-truth-reward training and other rubric-based training settings.

📊 Key Results

  • AppWorld 领域： DRACO 相比基础模型实现了 15.9 个百分点的提升，相比使用稀疏真实地面奖励训练的 GRPO 实现了 5.3 个百分点的提升（且不依赖验证器）。
  • Tau-Bench 领域（域外）： DRACO 在没有前沿评判器的情况下实现了 5.3 个百分点的提升，超越了标准真实奖励训练和替代的基于评分标准的方法。

📊 Key Results

  • AppWorld Domain: DRACO achieves a 15.9-point gain over the base model and a 5.3-point gain over GRPO trained with a sparse ground-truth reward (without relying on verifiers).
  • Tau-Bench Domain (Out-of-Domain): DRACO secures a 5.3-point gain over the base model without a frontier judge, outperforming both standard ground-truth-reward training and alternative rubric-based approaches.