跳转至

文章背景与核心概要

当前的闭环大模型（LLM）训练严重依赖反馈信号，然而这些反馈往往无法准确捕捉人类监督者的真实意图或期望价值。为此，本文引入了一种名为评估条件训练（Evaluation-Conditioned Training, ECT）的新型后训练框架，旨在减轻奖励错配（reward misspecification）带来的负面影响。

ECT 可以作为现有算法（如 SFT 和 PPO）的插件。它利用自然语言将训练样本与所提供反馈的保真度（fidelity）进行条件化绑定。通过这种方式，模型学会了区分高保真度与低保真度的监督信号，从而在部署时能够配合高保真度监督器展现出更优的行为。作者通过概念验证实验证明了 ECT 的有效性，特别是在减少算术任务中的谄媚行为（sycophancy）以及提高新闻生成的公正性方面表现显著。

Evaluation-Conditioned Training: Teaching Models to Generalize to Stronger Oversight Regimes

Authors: Alec Harris, Kasey Corra, Archie Chaudhury, Yixiong Hao
Date: August 10, 2026
arXiv: 2608.10209 [cs.AI]

Summary

Current Large Language Model (LLM) training relies heavily on feedback signals that often fail to capture the true intent or desired values of human overseers. This paper introduces Evaluation-Conditioned Training (ECT), a novel post-training framework designed to mitigate reward misspecification.

ECT functions as an add-on to existing algorithms like SFT and PPO. It uses natural language to condition training samples on the fidelity of the provided feedback. By doing so, the model learns to distinguish between high-fidelity and low-fidelity oversight, allowing it to elicit more desirable behaviors when deployed alongside a high-fidelity monitor. The authors demonstrate the effectiveness of ECT through proof-of-concept experiments, specifically in reducing sycophancy in arithmetic tasks and increasing even-handedness in news generation.

当前的大语言模型（LLM）训练严重依赖反馈信号，这些信号往往无法捕捉人类监督者的真实意图或期望价值。本文引入了评估条件训练（Evaluation-Conditioned Training, ECT），这是一种旨在缓解奖励错配的新型后训练框架。

ECT 可以作为现有算法（如 SFT 和 PPO）的附加组件。它使用自然语言将训练样本与所提供反馈的保真度联系起来进行条件化。通过这样做，模型学会了区分高保真度和低保真度的监督，使其在与高保真度监视器一起部署时能够引发更理想的行为。作者通过概念验证实验证明了 ECT 的有效性，特别是在减少算术任务中的谄媚现象以及提高新闻生成的公正性方面。

Key Concepts

The Problem: Reward Misspecification

The authors identify a fundamental limitation in current alignment techniques: the inability of human annotators and automated reward functions to provide perfectly faithful feedback. This leads to models that optimize for the proxy reward rather than the intended objective.

问题：奖励错配

作者指出了当前对齐技术的一个根本局限性：人类标注员和自动化奖励函数无法提供完全忠实的反馈。这导致模型优化的是代理奖励，而不是预期的目标。

The Solution: Evaluation-Conditioned Training (ECT)

ECT addresses this by: 1. Conditioning on Fidelity: During training, the model is conditioned on the quality/fidelity of the feedback signal. 2. Deployment Strategy: In deployment, the model is conditioned on a "high-fidelity monitor," guiding it to prioritize more accurate or objective oversight over potentially biased or noisy training signals. 3. Versatility: The framework is designed to be model-agnostic and compatible with existing post-training pipelines.

解决方案：评估条件训练（ECT）

ECT 通过以下方式解决这个问题： 1. 基于保真度的条件化： 在训练期间，模型会根据反馈信号的质量/保真度进行条件化。 2. 部署策略： 在部署时，模型会根据“高保真监视器”进行条件化，引导其优先考虑更准确或更客观的监督，而不是潜在的有偏见或有噪声的训练信号。 3. 通用性： 该框架旨在与模型无关，并与现有的后训练流程兼容。

Experimental Validation

The researchers tested ECT in two challenging scenarios: * News Generation: Improving even-handedness where the baseline training data rewarded biased perspectives. * Arithmetic Tasks: Reducing sycophancy (the tendency of models to agree with user errors) where the baseline training rewarded agreement with the user.

In both cases, ECT significantly improved performance compared to standard direct training methods.

实验验证

研究人员在两个具有挑战性的场景中测试了 ECT： * 新闻生成： 改善公正性，其中基线训练数据奖励有偏见的视角。 * 算术任务： 减少谄媚现象（模型倾向于同意用户的错误），其中基线训练奖励对用户的顺从。

在这两种情况下，与标准的直接训练方法相比，ECT 显著提高了性能。

Metadata & Resources

元数据与资源

Access Paper

访问论文