跳转至

文章背景与核心概要

在当前的科学研究和数据分析中，大语言模型（LLM）智能体正越来越多地被用于检查数据集、编写代码并执行端到端的自动化分析。然而研究发现，尽管这些智能体在技术执行层面往往表现正确，但由于它们经常忽视底层数据的基本假设，导致产出的 p 值在统计学上并不可靠，从而引发隐蔽的统计推断错误。

为了解决这一痛点，研究人员推出了 P-Bench 基准测试集，涵盖经济学、生物学和医学等领域的 425 个真实且开放的假设检验任务。在此基础上，他们通过合成任务与经验证的统计奖励，利用强化学习训练出了开源大模型智能体 Fisher-R1-14B。

实验结果表明，Fisher-R1-14B 的性能显著优于现有的尖端闭源和开源基准模型（包括 GPT-5.4 和 DeepSeek-V4-Pro），在单次试验成功率上实现了平均 21% 的相对提升，在最具挑战性的任务上甚至达到了 26% 的提升。这项研究证明了强化学习结合统计奖励在纠正大模型统计推理缺陷方面的巨大潜力。

Fisher-R1: Training LLM Agents for Reliable Hypothesis Testing

license icon

Summary

Fisher-R1 is an open-weight Large Language Model (LLM) agent engineered to address subtle inferential and statistical errors commonly made by AI during automated hypothesis testing. While LLM agents are increasingly used to inspect datasets, write code, and conduct end-to-end analyses, they frequently produce statistically invalid p-values despite correct technical executions because they ignore underlying data assumptions.

To evaluate and solve this problem, the researchers introduced: 1. P-Bench: A rigorous benchmark consisting of 425 open-ended, realistic hypothesis-testing tasks across economics, biology, and medicine. 2. Fisher-R1-14B: An open-weight model trained via reinforcement learning using synthetic tasks and verified statistical rewards.

Key Findings: Fisher-R1-14B significantly outperforms existing state-of-the-art proprietary and open-source baselines—including GPT-5.4 and DeepSeek-V4-Pro—achieving a 21% average relative improvement in single-trial success (and up to 26% on the most challenging tasks).

Document Metadata

Field Details
arXiv ID arXiv:2608.07437 [cs.AI]
Primary Subject Computer Science > Artificial Intelligence (cs.AI)
Submission Date August 7, 2026
Authors Jiacheng Miao, Jin Mu, Guanhua Chen, James Zou
DOI 10.48550/arXiv.2608.07437

Abstract

可靠的假设检验是许多实证科学论断的基石。大语言模型（LLM）智能体正日益被用于自动化这一过程，因为它们能够检查数据集、生成代码并端到端地产出分析结果。然而，我们发现它们经常犯下隐蔽的推断错误，尽管分析执行过程完全正确，却导致了不正确的结论。

现有基准无法捕捉到这种失效模式，因为它们极少评估在给定数据潜在假设的前提下，报告的 p 值在统计学上是否有效。为了填补这一空白，我们构建了 P-Bench，这是一个包含 425 个开放式、现实世界假设检验任务的基准，跨越了经济学、生物学和医学领域。每个任务都要求智能体仅根据科学假设和数据集来选择统计方法、计算 p 值并得出结论。

我们进一步推出了 Fisher-R1，这是一个通过合成任务和强化学习训练出来的开源大模型智能体，专门用于进行严谨的假设检验。在 P-Bench 上，Fisher-R1-14B 相比其基座模型有了实质性的提升，并且超越了包括 GPT-5.4 和 DeepSeek-V4-Pro 在内的强大闭源与开源基准。相较于 DeepSeek-V4-Pro，它在单次试验成功率上实现了平均 21% 的相对提升，在最具挑战性的任务上提升幅度甚至高达 26%。我们的结果表明，当前的大语言模型智能体在假设检验方面缺乏可靠的统计推理能力，而针对带有已验证统计奖励的任务进行强化学习能够大幅提升其可靠性。

Reliable hypothesis testing is the foundation of many empirical scientific claims. Large language model (LLM) agents are increasingly used to automate this process, as they can inspect datasets, generate code, and produce analyses end-to-end. However, we show that they frequently make subtle inferential errors that lead to incorrect conclusions despite correctly executed analyses.

Existing benchmarks fail to capture this failure mode, as they rarely assess whether a reported p-value is statistically valid given the assumptions underlying the data. We address this gap by building P-Bench, a benchmark comprising 425 open-ended, realistic hypothesis-testing tasks spanning economics, biology, and medicine. Each task requires an agent to select a statistical method, compute a p-value, and draw a conclusion given only a scientific hypothesis and a dataset.

We further introduce Fisher-R1, an open-weight LLM agent trained for rigorous hypothesis testing using synthetic tasks and reinforcement learning. On P-Bench, Fisher-R1-14B substantially improves over its backbone and outperforms strong proprietary and open-source baselines, including GPT-5.4 and DeepSeek-V4-Pro, achieving a 21% average relative improvement in single-trial success over DeepSeek-V4-Pro, with gains up to 26% on the most challenging tasks. Our results demonstrate that current LLM agents lack reliable statistical reasoning for hypothesis testing and that reinforcement learning on tasks with verified statistical reward substantially improves reliability.

访问论文与资源