跳转至

评测工具隐形错误如何颠覆智能体排行榜排名

arXiv:2611.04399 [cs.AI]
Subjects: Artificial Intelligence (cs.AI); Software Engineering (cs.SE)
Authors: Yangze Liu (Shandong University)
Submitted: 3 Nov 2026
DOI / Paper: 10.48550/arXiv.2611.04399
Project / Code: GitHub - YangzeLiu/eval-instrument-disclosure

文章背景与核心概要

在当前的 AI 研究与工程实践中，各大公开排行榜 (Leaderboard) 被广泛视作衡量大语言模型 (Large Language Model, LLM) 与 AI 智能体 (AI Agent) 能力高下的终极试金石。然而，极少有人怀疑“裁判”本身是否准确无误。山东大学研究人员 Yangze Liu 在这项研究中揭露了一个普遍却被忽视的行业现象：评测基建本身存在的代码漏洞、打分脚本缺陷和版本漂移，正在悄无声息地扭曲基准测试 (Benchmark) 的客观性。许多在榜单上排名垫底的智能体，并非推理能力不行，而是因为评测脚本根本没对它们的对话轮次打分并直接判负。本文通过对 HAL 和 tau2-bench 等主流榜单的深度审查，详尽阐述了这些“隐形工具错误”带来的严重偏差，并提出了四项极简却高鲁棒的校验准则。

核心概要

Summary

在论文《评测工具隐形错误如何颠覆智能体排行榜排名》中，来自山东大学的作者 Yangze Liu 深入调研了评测基础设施（如打分脚本、日志记录缺陷与评估器版本迭代）中的缺陷如何在完全不报错的情况下悄然扭曲 AI 智能体的性能指标。通过对 HAL 和 tau2-bench 等知名公开排行榜的严密审查，研究表明：未评分的对话轮次、重复计算的模型调用成本以及评估器内部逻辑漏洞，都会在没有触发任何报错警告的前提下，彻底颠覆智能体的排名位次。作者据此提出了一套轻量级的完整性校验机制，旨在保障智能体评测的真实可靠。

In the paper "Silent Instrument Errors Reorder Agent Leaderboards", author Yangze Liu (Shandong University) investigates how flaws in evaluation infrastructure—such as grading scripts, logging errors, and evaluator versioning—silently distort AI agent performance metrics. By auditing prominent leaderboards like HAL and tau2-bench, the study reveals that unscored episodes, miscounted model costs, and evaluator bugs can drastically alter agent rankings without any visible errors. The author proposes lightweight integrity checks to ensure more reliable agent evaluations.

论文元数据

Metadata

论文摘要

Abstract

在某公开智能体排行榜上，一个通用型智能体在涉及数据库操作的航空客服基准测试任务中，落后于专门针对该场景打造的专用脚手架整整 13 分。读者很可能会据此得出结论：“通用性牺牲了准确率”。然而，这一鸿沟的真正根源其实出自打分脚本 (Grader)。

On a public agent leaderboard, a general-purpose agent trails a scaffold built for an airline customer-service benchmark by 13 points on the tasks that touch its database, and a reader could conclude that generality costs accuracy. The gap comes from the grader.

通用智能体在结束对话时的交互模式，常常无法触发该基准测试预设的自动打分逻辑。结果导致它只有大约一半的会话轮次被真正评分，其余轮次全都被粗暴地标记为失败并静默归档。在无需重新运行任何智能体的情况下，仅根据基准测试自身的既定规则对未打分的后半部分轨迹补评分，二者之间的综合分差便彻底消除。

The general agent often ends its conversations in a way that never triggers grading, so only about half of its episodes were scored and the rest were stored as failures. Grading the ungraded half under the benchmark's own rule closes the pooled gap, without rerunning any agent.

我们仅依据公开发布的评测制品，对维护极为严谨的两个知名公开榜单——HAL 和 tau2-bench 进行了独立审计，又发现了另外三种典型的测量工具错误： * 成本统计失真：调用成本统计列中将嵌套的模型调用重复计算了两次，且在不同行之间分布不均； * 动作来源混淆：归档的执行轨迹中，将打分脚本自身注入的操作错误记录为智能体的自主动作； * 版本迁移颠覆指标：在更新后的评测器下重新为保存的轨迹打分，其本质已构成一次全新的物理测量——一次官方公告的合理修复导致整个榜单排名颠覆；而一处在读取四个文件时发生模式误判的漏洞，在没有打印任何报错的情况下，直接让对应模型的分数断崖式跌落了 59 到 82 分。

We audit two carefully maintained public leaderboards, HAL and tau2-bench, from their published artifacts alone and find three more instrument errors: * A cost column counts nested model calls twice, unevenly across rows. * Archived trajectories record grader-written actions as agent actions. * Re-grading saved trajectories under a newer evaluator is itself a new measurement: a correct and announced fix reorders a leaderboard, and a mode check that misreads four files collapses their scores by 59 to 82 points while printing no error.

以上每一种错误在最终输出时都呈现为一个看似完全合理的数字。HAL 的维护团队在五周内全数确认了我们指出的三项缺陷。这些案例共同表明：智能体评测的基础设施——即负责打分、记录日志、回放轨迹和管理评估器版本的底层代码——本身就是测量系统不可分割的一部分，且极易发生静默失效；因此，基准测试的得分和榜单排名可能在毫无警示的情况下发生剧烈位移。此类故障模式源于当前智能体评测流水线的构建方式，随着被测智能体的演进，这些评测流水线也在高速迭代，因此我们预计同类问题在整个行业广泛存在。

Each produced a plausible number, and the HAL maintainers confirmed their three within five weeks. Together the cases show that the infrastructure of agent evaluation, meaning the code that grades, logs, replays, and versions an evaluation, is part of the measurement and fails silently, so scores and orderings can move with no visible sign. These failure modes follow from how agent evaluation pipelines are built, and those pipelines change as fast as the agents they measure, so we expect more of them elsewhere.

作者建议的解决方案

Proposed Solutions

为彻底根治这些隐形测量误差，作者提出了四种高性价比的健全性检查机制，以暴露评测工具层面的潜在缺陷： 1. 公布打分覆盖比例 (Graded Fraction)：强制汇报实际被成功打分的轮次比例，避免未打分被暗中计为失败； 2. 从根追踪聚合成本 (Aggregate Cost from Root Spans)：统一从跟踪树的根节点计算模型调用开销，杜绝嵌套调用重复统计； 3. 标记每个动作的来源出处 (Provenance Tagging)：对轨迹中记录的每一个动作明确标记来源（是智能体输出还是打分器环境干预）； 4. 锁定评测器版本并附带重测差异 (Pin Evaluator Versions & Regrade Diff)：严格锁定评测器依赖版本，并在任何更新时公开发布重测差异对比表。

To combat these silent errors, the author proposes cheap checks that expose the instrument: 1. Report the graded fraction. 2. Aggregate cost from root spans. 3. Tag the provenance of every recorded action. 4. Pin evaluator versions with a published re-grade diff.

全文获取与音频解读链接

Access & Full-Text Links