跳转至

移动智能体评估中大模型裁判的基准测试

文章背景与核心概要

随着移动智能体基准测试日益依赖大语言模型（LLM）作为“裁判”来评估任务完成情况，这些自动化评估方法的可靠性在很大程度上尚未得到验证。本文提出了 MobileJudgeBench，这是一个旨在系统性评估移动智能体轨迹背景下“LLM作为裁判”（LLM-as-a-judge）方法的综合基准。

该研究分析了涵盖 6 个移动智能体基准、4 个智能体模型以及 68 个应用程序的 931 条人工标注轨迹。通过测试多种裁判方法和 LLM 后端，作者为自动化评估流程的有效性提供了关键见解，揭示了模型选择与评估质量之间的核心关联。

关键发现 (Key Findings)

1. 简洁性与复杂性的权衡

研究表明，复杂的裁判流程并不总是优于简单的方法。一种利用采样截图的基准裁判方法表现出极强的竞争力，其性能往往超过了更复杂、专门构建的方法。研究结论指出，LLM 后端的选择是决定裁判质量的主要驱动因素。

The research demonstrates that elaborate judge pipelines do not consistently outperform simpler methods. A baseline judge utilizing sampled screenshots proved to be highly competitive, often exceeding the performance of more complex, purpose-built methods. The study concludes that the choice of the LLM backbone is the primary driver of judge quality.

2. 预测效用

作者发现，基准质量指标是衡量现实世界效用的可靠预测因子。这些指标与以下方面高度相关： * 评估保真度 (Evaluation Fidelity)： 裁判准确评估智能体性能的能力。 * 下游性能 (Downstream Performance)： 当裁判作为策略强化学习的奖励信号使用时，其有效性。

The authors found that benchmark quality metrics are reliable predictors of real-world utility. These metrics correlate strongly with: * Evaluation Fidelity: The ability of the judge to accurately rank agent performance. * Downstream Performance: The effectiveness of the judge when used as a reward signal for on-policy reinforcement learning.

3. 失败特征分析

对不同 LLM 后端的详细故障分析揭示了独特且往往截然相反的行为模式： * 保守型与宽松型 (Conservative vs. Permissive)： 不同的后端表现出特定的精确率-召回率特征，导致了性质迥异的错误类型。理解这些特征对于为特定智能体评估任务选择合适的裁判至关重要。

A detailed failure analysis across different LLM backends revealed distinct, often opposing, behavioral patterns: * Conservative vs. Permissive: Different backends exhibit specific precision-recall characteristics that lead to qualitatively different types of errors. Understanding these profiles is essential for selecting the appropriate judge for specific agent evaluation tasks.

访问与资源 (Access & Resources)

  • View PDF: arXiv:2608.11434
  • HTML (Experimental): arXiv HTML Version
  • Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV)