跳转至

文章背景与核心概要

在智能体文档提取领域，“保真度”（即检查提取的值是否与源文本匹配）通常被认为是成功的黄金标准。然而，本文指出仅靠保真度是远远不够且具有潜在危险的。作者在评估内部提取服务的模型时发现了一个关键的失效模式：结构化输出约束静默禁用了模型的工具使用功能。尽管模型从未打开过实际的数据表，但它通过幻觉出看似合理的源文本来回答提示词，从而成功通过了传统的保真度检查。只有逐个工具的执行追踪才能揭示这种欺骗行为。

为了解决这一可观测性鸿沟，研究人员在一个包含 37 个声明的智能体基准测试中记录了每一次工具调用。基于这种调度级遥测数据，他们开发了两项核心工具：1. 基于规则的失败归因分类器；2. 严格依赖工具调用模式（而非提取值）的静默失败检测器。核心研究结果表明，静默失败检测器在 207 个干净的、通过保真度的提取中保持了零误报率，并成功恢复了所有植入的故障。此外，对已部署模型栈的测试表明，工具层主要驱动的是可移植性和可观测性，而非单纯的原始准确率——这证明了当文档规模超出标准上下文窗口时其终极价值。

Fidelity Is Not Enough: Dispatch-Level Instrumentation for Agentic Datasheet Extraction

Fidelity Is Not Enough: Dispatch-Level Instrumentation for Agentic Datasheet Extraction

Authors: Qing Ye, Meng-Hsuan Lin
Accepted at: EMNLP 2026 Industry Track (7 pages + appendices)
arXiv: arXiv:2608.28439 [cs.CL] | DOI: 10.48550/arXiv.2608.28439
Submitted: August 28, 2026

Authors: Qing Ye, Meng-Hsuan Lin
Accepted at: EMNLP 2026 Industry Track (7 pages + appendices)
arXiv: arXiv:2608.28439 [cs.CL] | DOI: 10.48550/arXiv.2608.28439
Submitted: August 28, 2026

📌 Summary

📌 Summary

In agentic document extraction, fidelity (checking whether an extracted value matches the source text) is typically considered the gold standard for success. However, this paper demonstrates that fidelity alone is dangerously insufficient.

In agentic document extraction, fidelity (checking whether an extracted value matches the source text) is typically considered the gold standard for success. However, this paper demonstrates that fidelity alone is dangerously insufficient.

The authors discovered a critical failure mode while qualifying models for an internal extraction service: a structured-output constraint silently disabled a model's tool use. Despite never opening the actual datasheet, the model successfully passed traditional fidelity checks by hallucinating plausible source text to answer the prompt. Only per-tool execution tracing exposed the deception.

The authors discovered a critical failure mode while qualifying models for an internal extraction service: a structured-output constraint silently disabled a model's tool use. Despite never opening the actual datasheet, the model successfully passed traditional fidelity checks by hallucinating plausible source text to answer the prompt. Only per-tool execution tracing exposed the deception.

To address this observability gap, the researchers logged every tool call across an agentic benchmark spanning 37 total claims. From this dispatch-level telemetry, they developed two core instruments: 1. A rule-based failure-attribution classifier 2. A silent-failure detector relying strictly on tool-call patterns rather than extracted values

To address this observability gap, the researchers logged every tool call across an agentic benchmark spanning 37 total claims. From this dispatch-level telemetry, they developed two core instruments: 1. A rule-based failure-attribution classifier 2. A silent-failure detector relying strictly on tool-call patterns rather than extracted values

Key findings reveal that the silent-failure detector maintains a zero false-positive rate across 207 clean, fidelity-passing extractions and successfully recovers all planted faults. Furthermore, testing across deployed model stacks indicates that the tool layer primarily drives portability and observability rather than raw accuracy—proving its ultimate value when documents scale beyond standard context windows.

Key findings reveal that the silent-failure detector maintains a zero false-positive rate across 207 clean, fidelity-passing extractions and successfully recovers all planted faults. Furthermore, testing across deployed model stacks indicates that the tool layer primarily drives portability and observability rather than raw accuracy—proving its ultimate value when documents scale beyond standard context windows.