跳转至

文章背景与核心概要

在大语言模型（LLM）的评测中，由多个模型组成的裁判团（Judge Panels）被广泛应用，然而先前的研究表明它们往往存在高度相关的错误。例如，一个由九个裁判组成的评审团通常只能产生相当于两个独立裁判的有效信息，传统的聚合方法对此也无能为力。

本文深入探讨了引入外部验证信号（如执行测试套件）的影响，并揭示了几个关键洞察：聚合依赖性诊断与条件决策效用解决的是不同的问题，标准的人群级别指标未能捕捉到验证真正发挥作用的地方。作者通过基础的多数表决算术证明，单张选票的替换只能改变一票之差的查询决策，而实证结果也证实了准确率的提升完全集中在这些关键查询上。通过应用多数派侧替换规则，不仅能显著提升整体准确率，还大幅降低了对昂贵验证信号的调用频率。

Blind to the Pivotal Vote: Aggregate Independence Metrics Miss Where Verification Actually Helps

Blind to the Pivotal Vote: Aggregate Independence Metrics Miss Where Verification Actually Helps

Authors: Yang Shu
Primary Subject: Artificial Intelligence (cs.AI)
arXiv Identifier: arXiv:2608.06940 [cs.AI]
Submitted: August 7, 2026
DOI: 10.48550/arXiv.2608.06940

Authors: Yang Shu
Primary Subject: Artificial Intelligence (cs.AI)
arXiv Identifier: arXiv:2608.06940 [cs.AI]
Submitted: August 7, 2026
DOI: 10.48550/arXiv.2608.06940

Abstract Summary

Abstract Summary

大语言模型（LLM）裁判团被广泛用于模型评估，然而先前的研究表明，它们饱受高度相关错误的困扰。例如，一个由九个裁判组成的裁判团通常只能提供相当于两个独立裁判的有效信息，且传统的聚合方法几乎无法弥补这一差距。

Large Language Model (LLM) judge panels are widely used for evaluation, yet previous studies show they suffer from heavily correlated errors. For instance, a panel of nine judges often yields only the effective information of two independent ones, and traditional aggregation methods do little to bridge this gap.

本文研究了引入外部验证信号（例如执行测试套件）的过程，并揭示了核心洞察： * 独立性与效用： 聚合依赖性诊断和条件决策效用解决的是不同的问题。标准的总体级别指标未能捕捉到验证真正发挥作用的地方。 * 关键投票的力量： 利用初等多数表决算术，作者证明了单张选票的替换只能改变一票之差（one-vote margin）的查询决策。 * 集中的收益： 跨越三个代码基准测试和四种裁判团规模的实证发现证实，准确率的全部收益完全集中在这些关键查询上（在核心配置中带来了 +10.4 到 +23.3 个百分点的大幅跃升），而在其他地方没有任何收益。 * 高效的替换规则： 在 HumanEval+/MBPP+ 等基准测试中，应用多数派侧替换规则（majority-side replacement rule）可将整体准确率从 82.44% 提高到 85.62%，同时仅在 16.2% 的查询上调用了二级验证信号。

This paper investigates the introduction of external verification signals (such as executing a test suite) and reveals key insights: * Independence vs. Utility: Aggregate dependence diagnostics and conditional decision utility address different questions. Standard population-level metrics fail to capture where verification actually helps. * The Power of the Pivotal Vote: Using elementary majority arithmetic, the author demonstrates that single-ballot substitutions can only alter decisions on queries with a one-vote margin. * Concentrated Gains: Empirical findings across three code benchmarks and four panel sizes confirm that entire accuracy gains concentrate exclusively on these pivotal queries (yielding large jumps of +10.4 to +23.3 percentage points across headline configurations), with zero gains elsewhere. * Efficient Replacement Rules: On benchmarks like HumanEval+/MBPP+, applying a majority-side replacement rule raises overall accuracy from 82.44% to 85.62% while invoking the secondary verification signal on only 16.2% of queries.