跳转至

文章背景与核心概要

在科学事实核查领域，由大语言模型（LLM）驱动的系统通常仅根据最终裁决（如“正确”或“错误”）来评估声明。然而，仅仅匹配最终结论无法揭示AI系统是遵循与人类专家相同的严谨推理路径，还是通过表面正确实则存在缺陷的过程得出结论。本文引入了一种基于图的框架（类型化推理图），将解释映射为结构化图谱，从而能够深入比较人类与大模型在生物医学错误信息核查中的推理机制。

通过对MISSCIPLUS数据集中的84个虚假生物医学声明应用该方法，作者评估了GPT-5、Claude Opus 4.7和Qwen3-32B等主流模型。研究结果揭示了模型在裁决可靠性与人类对齐推理之间的独特权衡，为未来构建更具可解释性且符合人类逻辑的科学事实核查系统提供了关键洞见。

Beyond Verdicts: A Graph-Based Analysis of Human and LLM Reasoning in Scientific Fact-Checking

  • Authors: Abdul Ghafoor, Muhammad Arslan Manzoor, Yufang Hou
  • Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
  • arXiv ID: arXiv:2608.23047
  • Submitted: August 24, 2026

Beyond Verdicts: A Graph-Based Analysis of Human and LLM Reasoning in Scientific Fact-Checking

  • Authors: Abdul Ghafoor, Muhammad Arslan Manzoor, Yufang Hou
  • Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
  • arXiv ID: arXiv:2608.23047
  • Submitted: August 24, 2026

Executive Summary

基于大语言模型（LLM）的科学事实核查通常仅根据最终裁决（例如正确或错误）来评估声明。然而，仅匹配最终结论无法揭示AI系统是通过与人类专家相同的严谨推理路径得出决策，还是通过表面上正确但存在缺陷的过程得出决策。

本文引入了一种基于图的框架（类型化推理图，typed reasoning graph），将解释映射为结构化图谱，连接了错误声明、研究背景、实际发现、谬误支持前提以及谬误标签。通过将此方法应用于MISSCIPLUS数据集中的84个错误生物医学声明，作者评估了GPT-5、Claude Opus 4.7和Qwen3-32B等模型，揭示了裁决可靠性与人类对齐推理之间的独特权衡。

Executive Summary

Scientific fact-checking powered by Large Language Models (LLMs) typically evaluates claims based solely on a final verdict (e.g., Correct or Incorrect). However, matching final conclusions fails to reveal whether an AI system arrives at its decision through the same rigorous reasoning path as human experts or via a superficially correct yet flawed process.

This paper introduces a graph-based framework (typed reasoning graph) that maps explanations into structured graphs connecting false claims, study contexts, actual findings, fallacy-supporting premises, and fallacy labels. By applying this methodology to 84 false biomedical claims from the MISSCIPLUS dataset, the authors evaluate models such as GPT-5, Claude Opus 4.7, and Qwen3-32B, uncovering distinct trade-offs in verdict reliability versus human-aligned reasoning.

Abstract

引述合法论文的错误信息往往具有极大的危害性，特别是当它们扭曲了这些研究所实际报告的内容时。虽然现有的基于大语言模型（LLM）的自动事实核查系统可以评估模型是否给出了错误的裁决，并为该决定生成解释，但它们通常无法指明模型是遵循与人类专家相同的推理路径，还是通过另一条虽不同但仍然有效的路径得出裁决。

在这项工作中，我们引入了一个基于图的框架（类型化推理图），用于比较科学事实核查中人类和LLM的推理路径。基于先前关于生物医学错误信息中谬误推理的研究 MISSCIPLUS（Glockner 等人，2025年），我们将每个解释建模为一个推理图，该图将虚假声明与相关的研究背景、研究发现、谬误支持前提以及谬误标签联系起来。这种表示方法使得人类和LLM推理能够在谬误特定子图的层面上进行一对一对齐。

对于非人类对齐的LLM路径，我们验证了其在引用研究中的扎根性（grounding）、与声明的相关性（relevance）以及对裁决的充分性（sufficiency）。使用MISSCIPLUS中的84个虚假声明，我们在不同的提示词和证据设置下评估了GPT-5、Claude Opus 4.7和Qwen3-32B。结果显示出截然不同的性能维度：Qwen3-32B的裁决失败率最低，GPT-5的人类对齐度最高，而Claude Opus 4.7的裁决预测较弱，但在成功案例中通常具有有效的推理。

Abstract

Misinformation that cites legitimate papers can be especially harmful when it distorts what those studies actually report. While existing automatic fact-checking systems based on large language models (LLMs) can assess whether a model assigns an incorrect verdict and can generate explanations for that decision, they typically do not indicate whether the model follows the same reasoning path as human experts or arrives at the verdict through a different but still valid path.

In this work, we introduce a graph-based framework (typed reasoning graph) for comparing human and LLM reasoning paths in scientific fact-checking. Building on prior work on fallacious reasoning in biomedical misinformation, MISSCIPLUS (Glockner et al., 2025), we model each explanation as a reasoning graph that links the false claim to the relevant study context, study findings, fallacy-supporting premises, and fallacy labels. This representation enables one-to-one alignment of human and LLM reasoning at the level of fallacy-specific sub-graphs.

For non-human-aligned LLM paths, we validate grounding in the cited study, relevance to the claim, and sufficiency for the verdict. Using 84 false claims from MISSCIPLUS, we evaluate GPT-5, Claude Opus 4.7, and Qwen3-32B across prompt and evidence settings. Results show distinct performance dimensions: Qwen3-32B has the lowest verdict failure rate, GPT-5 the highest human alignment, and Claude Opus 4.7 weak verdict prediction but often valid reasoning in successful cases.

Methodology Highlights

  • 类型化推理图（Typed Reasoning Graphs）： 将解释建模为结构化网络，连接声明、背景、发现和特定的谬误标签。
  • 一对一对齐（One-to-One Alignment）： 实现了人类专家推理路径与自动化LLM输出之间直接、细粒度的比较。
  • 验证标准（Validation Criteria）： 非人类对齐的路径将根据以下标准进行严格评估：
  • 在引用的研究中的扎根性（Grounding）。
  • 与所分析声明的相关性（Relevance）。
  • 支持裁决的充分性（Sufficiency）。

Methodology Highlights

  • Typed Reasoning Graphs: Models explanations as structured networks linking claims, contexts, findings, and specific fallacy labels.
  • One-to-One Alignment: Enables direct, granular comparisons between human expert reasoning paths and automated LLM outputs.
  • Validation Criteria: Non-human-aligned paths are rigorously assessed based on:
  • Grounding in the cited study.
  • Relevance to the analyzed claim.
  • Sufficiency to support the verdict.

Key Findings

在84个生物医学错误信息案例中对 GPT-5、Claude Opus 4.7 和 Qwen3-32B 的评估表明，它们表现出截然不同的性能特征：

  • Qwen3-32B： 实现了最低的裁决失败率。
  • GPT-5： 表现出最高程度与人类专家推理路径的一致性。
  • Claude Opus 4.7： 表现出较弱的整体裁决预测能力，但在成功时往往采用逻辑上有效的推理。

Key Findings

Evaluating GPT-5, Claude Opus 4.7, and Qwen3-32B across 84 biomedical misinformation cases demonstrated distinct performance profiles:

  • Qwen3-32B: Achieved the lowest verdict failure rate.
  • GPT-5: Exhibited the highest degree of alignment with human expert reasoning paths.
  • Claude Opus 4.7: Demonstrated weaker overall verdict prediction, though it frequently employed logically valid reasoning when successful.

Full-Text and Resources

Full-Text and Resources