跳转至

文章背景与核心概要

在训练多轮工具调用大语言模型（LLM）智能体时，基于可验证奖励的强化学习（RLVR）提供了一个受验证器约束的性能上限，但它往往面临轨迹级别的信用分配问题，容易将不同轮次中异质的结果混淆成单一的奖励信号。相比之下，在线蒸馏（On-policy distillation）虽然能提供密集的逐词元（per-token）监督，但要么受限于教师模型的上限，要么容易陷入梯度集中崩塌。

为了克服这些局限性，本文作者引入了 \(\textbf{CrEST}\) 这一分层信用分配框架。该框架在保持强化学习验证器上限的同时，融入了来自特权自教师（self-teacher）的密集词元级信号。\(\textbf{CrEST}\) 在两个不同层面上解决信用分配问题：1. 通过分轮次验证优势（turn-segmented verified advantages）解决轮次间的稀释问题；2. 通过熵门控自教师调制（entropy-gated self-teacher modulation）精细化调整轮次内的词元贡献。

在 BFCL V3 和 WildToolBench 上的实验评估表明，在多种模型规模下，\(\textbf{CrEST}\) 的表现均持续优于标准的强化学习和蒸馏基线，尤其在长轨迹和严格的会话级指标上取得了显著增益。这项工作最终表明，教师在策略优化中的角色可以从“指示更新方向”转变为“调制更新幅度”，从而在不牺牲验证器上限的前提下实现密集的信用分配。

Teach the Magnitude, Not the Direction: Verifier-Bounded Credit Assignment for Multi-Turn Multi-step LLM Agents

Metadata Details
arXiv ID 2608.13179 [cs.AI]
Subjects Artificial Intelligence (cs.AI)
Submitted August 13, 2026
Authors Zechuan Wang, Siyuan Lu, Hongxuan Zhang, Linjian Mo, Chenyi Zhuang, Leilei Gan
Links View PDF | TeX Source

Summary

使用带可验证奖励的强化学习（RLVR）训练多轮工具使用 LLM 智能体，可以提供一个由验证器确定的性能上限，但它遭遇了轨迹级别的信用分配困境，将异质的单轮结果混淆为一个单一的奖励信号。相反，在线蒸馏提供了稠密的逐词元监督，但要么受限于教师模型的上限，要么容易发生梯度集中崩塌。

为了克服这些局限性，作者引入了 \(\textbf{CrEST}\)，这是一个分层的信用分配框架，它保留了强化学习由验证器界定的性能上限，同时结合了来自特权自教师的稠密词元级信号。\(\textbf{CrEST}\) 在两个不同层面上解决信用分配问题： 1. 轮次分割的验证优势（Turn-segmented verified advantages），以解决轮次间的信用稀释。 2. 熵门控自教师调制（Entropy-gated self-teacher modulation），以精细化单轮内部词元的贡献。

在 BFCL V3 和 WildToolBench 上的实验评估表明，\(\textbf{CrEST}\) 在多个模型规模下均稳定超越了标准强化学习和蒸馏基线，尤其在长轨迹和严格的会话级指标上取得了极强的性能增益。归根结底，这项工作表明，教师在策略优化中的角色可以从“决定更新方向”转变为“调制更新幅度”，从而在不牺牲验证器界定上限的前提下，解锁稠密的信用分配。

Training multi-turn tool-use LLM agents using Reinforcement Learning with Verifiable Rewards (RLVR) provides a verifier-bounded performance ceiling, but suffers from trajectory-level credit assignment that conflates heterogeneous per-turn outcomes into a single reward signal. Conversely, on-policy distillation provides dense per-token supervision, but remains either teacher-bounded or vulnerable to gradient concentration collapse.

To overcome these limitations, the authors introduce \(\textbf{CrEST}\), a hierarchical credit assignment framework that preserves the verifier-bounded ceiling of RL while integrating dense token-level signals from a privileged self-teacher. \(\textbf{CrEST}\) resolves credit at two distinct levels: 1. Turn-segmented verified advantages to resolve inter-turn dilution. 2. Entropy-gated self-teacher modulation to refine intra-turn token contributions.

Experimental evaluations on BFCL V3 and WildToolBench demonstrate that \(\textbf{CrEST}\) consistently outperforms both standard RL and distillation baselines across multiple model scales, achieving particularly strong gains on long-trajectory and strict session-level metrics. Ultimately, this work illustrates that a teacher's role in policy optimization can be shifted from dictating update directions to modulating update magnitudes, unlocking dense credit assignment without compromising the verifier-bounded performance ceiling.

Abstract

强化学习与可验证奖励（RLVR）为训练多轮工具使用智能体提供了一个由验证器界定的性能上限，然而其轨迹级别的信用分配将异质的单轮结果混淆为了单一的奖励信号。在线蒸馏提供了密集的逐词元监督，但要么受限于教师模型的上限，要么容易发生梯度集中崩塌。我们引入了 \(\textbf{CrEST}\)，这是一个分层的信用分配框架，它保留了强化学习的验证器界定上限，同时融合了来自特权自教师的密集词元级信号。\(\textbf{CrEST}\) 在两个层面上解决信用分配：轮次分割的验证优势解决了轮次间稀释，而熵门控自教师调制则精细化了单轮内的词元贡献。在 BFCL V3 和 WildToolBench 上的实验表明，\(\textbf{CrEST}\) 在两种模型规模下均持续优于强化学习和蒸馏基线，其中在长轨迹和严格的会话级指标上增益最大。我们的工作表明，教师在策略优化中的角色可以从决定更新方向简化为调制更新幅度，从而在不牺牲验证器上限的前提下解锁密集的信用分配。

Reinforcement learning with verifiable rewards (RLVR) offers a verifier-bounded performance ceiling for training multi-turn tool-use agents, yet its trajectory-level credit assignment conflates heterogeneous per-turn outcomes into a single reward signal. On-policy distillation provides dense per-token supervision but is either teacher-bounded or prone to gradient concentration collapse. We introduce \(\textbf{CrEST}\), a hierarchical credit assignment framework that retains RL's verifier-bounded ceiling while incorporating dense token-level signals from a privileged self-teacher. \(\textbf{CrEST}\) resolves credit at two levels: turn-segmented verified advantages address inter-turn dilution, while entropy-gated self-teacher modulation refines intra-turn token contributions. Experiments on BFCL V3 and WildToolBench show that \(\textbf{CrEST}\) consistently outperforms both RL and distillation baselines across two model scales, with the largest gains on long-trajectory and strict session-level metrics. Our work demonstrates that the teacher's role in policy optimization can be reduced from determining update directions to modulating update magnitudes, unlocking dense credit assignment without sacrificing the verifier-bounded ceiling.

Associated Resources & Tools