跳转至

FARCA：面向具有事实监督的强化学习的事实对齐可靠性感知信用分配

文章背景与核心概要

在大语言模型（LLMs）的强化学习训练中，采用基于可验证奖励的方法常常会由于结果导向的奖励机制而引发幻觉风险。尽管现有的缓解策略引入了过程级的事实监督，但它们通常因“事实信号的粗粒度聚合”和“缺乏可靠性评估”而存在不足，这导致了事实验证与策略更新之间的错配——作者将这种现象称为噪声事实信用分配（noisy factual credit assignment）。

为了解决这一问题，本文提出了 FARCA（Fact-Aligned Reliability-Aware Credit Assignment，事实对齐可靠性感知信用分配）这一新型策略优化框架。FARCA 通过解决两个核心模糊性问题，将事实监督转化为局部化、带可靠性权重的 Token 级训练信号：1. 信用定位模糊性（通过细粒度的信用定位，将事实验证的粒度与策略更新相匹配来解决）；2. 信用可靠性模糊性（通过反事实证据归因来解决，该方法利用事实判断对关键证据的依赖性来经验性地计算可靠性权重）。通过这些机制，FARCA 有效地调节了事实奖励和局部策略优势，最大程度地减小了不可靠信号的影响，在保持通用推理能力的同时，显著提升了模型在多个基准测试中的事实准确性。

执行摘要

Large language models (LLMs) trained via reinforcement learning with verifiable rewards often suffer from hallucination risks driven by outcome-centric rewards. While existing mitigation strategies incorporate process-level factual supervision, they typically fall short due to coarse-grained aggregation of factual signals and a lack of reliability assessments, leading to a mismatch between fact verification and policy updates—a phenomenon the authors term noisy factual credit assignment.

通过具有可验证奖励的强化学习训练的大语言模型（LLMs），常常会遭受由结果中心型奖励驱动的幻觉风险。虽然现有的缓解策略纳入了过程级的事实监督，但由于事实信号的粗粒度聚合和缺乏对这些信号的可靠性评估，它们通常效果不佳，从而导致事实验证与策略更新之间产生错配——作者将这一现象称为噪声事实信用分配。

To resolve this, the paper introduces FARCA (Fact-Aligned Reliability-Aware Credit Assignment), a novel policy optimization framework. FARCA transforms factual supervision into localized, reliability-weighted token-level training signals by addressing two key ambiguities: 1. Credit localization ambiguity, resolved through fine-grained credit localization that matches the granularity of fact verification with policy updates. 2. Credit reliability ambiguity, resolved via counterfactual evidence attribution, which leverages the dependence of factual judgments on key evidence to empirically compute reliability weights.

为了解决这一问题，本文引入了 FARCA（Fact-Aligned Reliability-Aware Credit Assignment，事实对齐可靠性感知信用分配），这是一个新颖的策略优化框架。FARCA 通过解决两个关键的模糊性问题，将事实监督转化为局部化的、带有可靠性权重的 Token 级训练信号： 1. 信用定位模糊性：通过细粒度的信用定位来解决，该定位将事实验证的粒度与策略更新的粒度相匹配。 2. 信用可靠性模糊性：通过反事实证据归因来解决，该方法利用事实判断对关键证据的依赖性，经验性地计算可靠性权重。

Through these mechanisms, FARCA effectively modulates factual rewards and local policy advantages, minimizing the impact of unreliable signals and significantly enhancing model factuality across multiple benchmarks while preserving general reasoning capabilities.

通过这些机制，FARCA 有效地调节了事实奖励和局部策略优势，最大限度地减少了不可靠信号的影响，并在保持通用推理能力的同时，显著增强了模型在多个基准测试中的事实准确性。

摘要

To reduce the hallucination risk caused by outcome-driven rewards in large language models trained through reinforcement learning with verifiable rewards, existing mitigation approaches introduce process-level factual supervision. However, due to coarse-grained aggregation of factual signals and the lack of reliability assessment for these signals, they create a mismatch between fact verification and policy updates. We term this noisy factual credit assignment and decompose it into two aspects: credit localization ambiguity and credit reliability ambiguity. To address these issues, we propose FARCA (Fact-Aligned Reliability-Aware Credit Assignment), a policy optimization framework that transforms factual supervision into localized, reliability-weighted token-level training signals. FARCA achieves fine-grained credit localization by aligning the granularity of fact verification with that of policy updates. It further introduces counterfactual evidence attribution, which uses the dependence of a factual judgment on key evidence as an empirical proxy for verification reliability to compute reliability weights. These weights modulate factual rewards and local policy advantages, reducing the influence of potentially unreliable signals on policy optimization. Experiments across different models and multiple factual reasoning benchmarks show that FARCA significantly improves model factuality while preserving general reasoning capabilities.

为了减少通过具有可验证奖励的强化学习训练的大语言模型中由结果驱动的奖励所带来的幻觉风险，现有的缓解方法引入了过程级的事实监督。然而，由于事实信号的粗粒度聚合以及对这些信号缺乏可靠性评估，它们导致了事实验证与策略更新之间的错配。我们将此称为噪声事实信用分配，并将其分解为两个方面：信用定位模糊性和信用可靠性模糊性。为了解决这些问题，我们提出了 FARCA（事实对齐可靠性感知信用分配），这是一个策略优化框架，它将事实监督转化为局部化的、加权可靠性的 Token 级训练信号。FARCA 通过将事实验证的粒度与策略更新的粒度相匹配，实现了细粒度的信用定位。它进一步引入了反事实证据归因，该方法利用事实判断对关键证据的依赖性作为验证可靠性的经验代理，以计算可靠性权重。这些 weights 调节了事实奖励和局部策略优势，减少了潜在不可靠信号对策略优化的影响。在不同模型和多个事实推理基准上的实验表明，FARCA 在保持通用推理能力的同时，显著提高了模型的事实准确性。

获取论文与资源

license icon