无真实标签的信用分配：基于执行重放对大模型智能体步级信用分配进行审计
文章背景与核心概要
在大语言模型（LLM）智能体的研究中，如何准确评估和分配每一步动作对最终结果的贡献（即步级信用分配，Step-level Credit Assignment）一直是一个核心难题。由于缺乏直接的真实标签（Ground Truth），现有的方法通常依赖LLM裁判打分、结果条件对数概率比（Logprob Ratios）或策略自身的置信度来估计每一步的好坏。然而，这些代理指标真的可靠吗？
本文作者对LLM智能体的步级信用分配信号进行了严谨的审计。通过在ALFWorld单智能体工具环境中将这些信用信号与执行重放（Executed Replay）中得出的策略条件真实标签进行对比，研究发现：现有的标准信用度量指标无法提供比打乱控制（Shuffled Controls）更可靠的增量保真度。隐式信用仅仅是在复现文本的流畅度，而结果条件反射则未能提供任何因果信息。此外，预注册的训练实验表明，没有哪种测试过的信用分配策略能够稳定地跑赢未训练的策略，信用性能的差异往往只是有效训练剂量（样本量）的人为产物，而非信用信号本身的优劣。该研究为智能体强化学习和信用分配机制带来了深度的反思与警示。
📌 Summary
本文探讨了大型语言模型（LLM）智能体中步级信用分配信号的可靠性。通过在ALFWorld中将这些信号与执行重放（executed replays）中的策略条件真实标签进行对比，作者发现：标准指标——如LLM裁判评分、结果条件对数概率比以及策略置信度——无法提供超越打乱控制（shuffled controls）的可靠增量保真度。
主要结论包括： * 正确性与贡献度的鸿沟： 尽管信号通常根据步骤的正确性进行评分，但通过重新采样反事实并向前滚动（rolling forward）来针对步骤的贡献度进行审计时，却暴露出了根本性的缺陷。 * 可识别的失效模式： 隐式信用仅仅呼应了文本的流畅度（中位秩相关系数达 +0.75），而结果条件化则未能贡献因果信息。 * 训练与样本量陷阱： 在一项预注册的训练实验中，没有一种测试过的信用分配策略能够可靠地优于未训练的策略。该研究强调，信用表现上的差异往往是有效训练剂量（样本量）的人为产物，而非信用信号本身。
This paper investigates the reliability of step-level credit assignment signals in Large Language Model (LLM) agents. Evaluating these signals against policy-conditional ground truth from executed replays in ALFWorld, the author discovers that standard metrics—such as LLM-judge scores, outcome-conditioned logprob ratios, and policy confidence—fail to provide reliable incremental fidelity over shuffled controls.
Key takeaways include: * The Correctness vs. Contribution Gap: While signals are typically graded on step correctness, auditing them against step contribution (via re-sampling counterfactuals and rolling forward) reveals fundamental flaws. * Identifiable Failure Modes: Implicit credit merely echoes text fluency (median rank correlation of +0.75), whereas outcome conditioning fails to contribute causal information. * Training and Sample Size Pitfalls: In a pre-registered training experiment, no tested credit allocation strategy reliably outperformed an untrained policy. The study highlights that discrepancies in credit performance are frequently artifacts of effective training dose (sample size) rather than the credit signal itself.
📑 Metadata & Reference
- 评论： 52页，6张图表。经过预注册，附录中包含冻结的分析计划和提示词。
Comments: 52 pages, 6 figures. Pre-registered, with frozen analysis plans and prompts included in the appendices.
- Primary Links: View PDF | HTML Version | TeX Source
📝 Abstract
在单一智能体工具环境（ALFWorld）中，通过对执行重放的策略条件真实标签进行审计，我们发现所审计的步级信用信号——LLM裁判评分、结果条件对数概率比或策略自身的置信度——均未能在其自身的边缘匹配打乱控制之外展现出可靠的增量保真度。在校正了重放目标可靠性后，隐式保真度被限制在接近零的水平，而在达到的目标可靠性下，裁判保真度则不具有决定性。现有的评估通常根据带注释的步骤正确性对这些信号进行打分；而我们则是针对步骤的贡献度对其进行审计——即在每个决策点重新采样策略自身的替代方案并向前滚动会对结果产生什么影响——结果表明这两者并不一致。真实标签具有结构性：在其有定义的决策点中，有 30.5% 在实现的采样分辨率下表现出非零的重放对照，且可测量性取决于模型——在两个规模相似的策略之间，无策略支持的反事实点比例相差两倍（13.1% 对 26.8%）。该失效模式是可识别的：隐式信用呼应了策略的流畅度（中位秩相关系数 +0.75，在修正工具下的第二个模型族中以 +0.70 重现），而结果条件化则没有添加任何因果信息（偏相关系数 -0.004，Qwen）。仅靠置信度的路由器在随机水平上恢复了关键步骤，但每轮减少了 13.1% 的裁判成本（每条轨迹减少 14.0%）。在包含七个分支的预注册训练实验中，没有任何一个分支能够可靠地优于未训练的策略，并且检查点的表观仪器特征在统计上与该设计中的有效训练剂量中介作用相一致——更稀疏的信用保留了更少的样本，优化器步骤存在一个数量级的差距——而不是信用内容本身。信用规则的比较必须匹配有效样本量，否则它们度量的是剂量而非信用。
Audited against policy-conditional ground truth from executed replay in a single-agent tool environment (ALFWorld), none of the step-level credit signals we audit -- LLM-judge scores, outcome-conditioned logprob ratios, or the policy's own confidence -- shows reliable incremental fidelity beyond its own marginal-matched shuffled control. Correcting for replay-target reliability leaves implicit fidelity bounded near zero and judge fidelity inconclusive at the achieved target reliability. Existing evaluations grade these signals against annotated step correctness; we audit them against step contribution -- what re-sampling the policy's own alternatives at each decision point and rolling forward changes about the outcome -- and they come apart. The ground truth is structured: 30.5% of decision points where it is defined exhibit a nonzero replay contrast at the achieved sampling resolution, and measurability is model-dependent -- the fraction of points with no policy-supported counterfactual differs twofold (13.1% vs. 26.8%) between two similar-scale policies. The failure mode is identifiable: implicit credit echoes the policy's fluency (median rank correlation +0.75, replicating at +0.70 in a second family under a corrected instrument), while outcome conditioning adds no causal information (partial correlation -0.004, Qwen). A confidence-only router recovers pivotal steps at chance level, but cuts judge cost by 13.1% per turn (14.0% per trajectory). In a seven-arm pre-registered training experiment, no arm reliably outperforms the untrained policy, and the checkpoints' apparent instrument signature is statistically consistent with mediation by effective training dose in this design -- sparser credit retains fewer examples, an order-of-magnitude spread in optimizer steps -- not credit content. Comparisons of credit rules must match effective sample size, or they measure dose, not credit.