文章背景与核心概要

大语言模型的基准测试结果极大地影响着厂商的市场声誉和商业估值。然而，当前的评估框架在很大程度上依赖于一种“荣誉系统”，饱受厂商偏见、未披露的模型更新、数据污染，以及“大模型即裁判”（LLM-as-a-judge）缩放方法中存在的身份感知偏见等问题的困扰。

本文针对七个主流裁判模型在匿名与身份公开问题上的表现，对评估偏见进行了深入研究。为了解决这些信任危机，作者提出了一种基于以太坊兼容账本的、利用自主经济代理（Autonomous Economic Agents）的区块链提交-揭晓协议（commit-reveal protocol），从而为构建值得信赖的去中心化AI基准测试建立了防篡改的审计追踪机制。

Who Verifies the Benchmark? Decentralizing Trust in Large Language Model Evaluation

arXiv ID: [arXiv:2608.07762 [cs.AI]] (https://arxiv.org/abs/2608.07762)
Authors: Sahil Pardasani, Madhusudan Singh
Accepted Venue: International Conference on Quantum Enhanced AI and Secure Computing (QASC2026)
Submitted: August 7, 2026

arXiv ID: [arXiv:2608.07762 [cs.AI]] (https://arxiv.org/abs/2608.07762)

Authors: Sahil Pardasani, Madhusudan Singh

Accepted Venue: International Conference on Quantum Enhanced AI and Secure Computing (QASC2026)

Submitted: August 7, 2026

📌 Executive Summary

Large Language Model (LLM) benchmarks heavily influence market reputations and commercial valuations. However, current evaluation frameworks often rely on an "honor system," plagued by vendor bias, undisclosed model updates, data contamination, and identity-aware bias in "LLM-as-a-judge" scaling methods.

This paper investigates evaluation bias across seven major verifier models tested on anonymous versus identity-disclosed questions. To resolve these trust issues, the authors propose a blockchain-based commit-reveal protocol using Autonomous Economic Agents on an Ethereum-compatible ledger, establishing a tamper-evident audit trail for trustworthy, decentralized AI benchmarking.

📌 Executive Summary

Large Language Model (LLM) benchmarks heavily influence market reputations and commercial valuations. However, current evaluation frameworks often rely on an "honor system," plagued by vendor bias, undisclosed model updates, data contamination, and identity-aware bias in "LLM-as-a-judge" scaling methods.

This paper investigates evaluation bias across seven major verifier models tested on anonymous versus identity-disclosed questions. To resolve these trust issues, the authors propose a blockchain-based commit-reveal protocol using Autonomous Economic Agents on an Ethereum-compatible ledger, establishing a tamper-evident audit trail for trustworthy, decentralized AI benchmarking.

🔍 Key Background & Problem Statement

The Trust Crisis: Vendor benchmarks lack strict cryptographic verification. For example, unverified claims surrounding DeepSeek R1 outperforming OpenAI’s o1 contributed to massive market volatility (including Nvidia losing USD 589 billion on January 27, 2025).
Flaws in Current Methods:
Independent leaderboards frequently uncover undisclosed proprietary model adjustments and contaminated training datasets.
Automated "LLM-as-a-judge" approaches suffer from identity-aware bias, scoring outputs based on the source model's prestige rather than objective quality.

🔍 Key Background & Problem Statement

The Trust Crisis: Vendor benchmarks lack strict cryptographic verification. For example, unverified claims surrounding DeepSeek R1 outperforming OpenAI’s o1 contributed to massive market volatility (including Nvidia losing USD 589 billion on January 27, 2025).

Flaws in Current Methods:

Independent leaderboards frequently uncover undisclosed proprietary model adjustments and contaminated training datasets.

Automated "LLM-as-a-judge" approaches suffer from identity-aware bias, scoring outputs based on the source model's prestige rather than objective quality.

🧪 Experimental Methodology & Findings

The study tested seven verifier models—GPT-OSS 120B, Llama 3.3 70B, GLM 5.1, Qwen3 32B, DeepSeek V4 Pro, Mistral Large3, and Sarvam M—evaluating both anonymous and identity-disclosed responses from three primary models across 58 factual, reasoning, political, and preference-based tasks.

Core Observations:

Factual & Reasoning Tasks: Identity disclosure slightly elevated scores for factual queries and moderately influenced stress-reasoning evaluations.
Geopolitical/Sensitive Topics: Revealing candidate identities triggered significant score shifts.
Notable Verifier Biases: Significant statistical anomalies occurred with specific models, such as:
GLM 5.1: \(+7.00\) points (\(p = 0.0249\))
Llama 3.3 70B: \(+1.56\) points (\(p = 0.00\))

🧪 Experimental Methodology & Findings

The study tested seven verifier models—GPT-OSS 120B, Llama 3.3 70B, GLM 5.1, Qwen3 32B, DeepSeek V4 Pro, Mistral Large3, and Sarvam M—evaluating both anonymous and identity-disclosed responses from three primary models across 58 factual, reasoning, political, and preference-based tasks.

Core Observations:

Factual & Reasoning Tasks: Identity disclosure slightly elevated scores for factual queries and moderately influenced stress-reasoning evaluations.

Geopolitical/Sensitive Topics: Revealing candidate identities triggered significant score shifts.

Notable Verifier Biases: Significant statistical anomalies occurred with specific models, such as:

GLM 5.1: \(+7.00\) points (\(p = 0.0249\))

Llama 3.3 70B: \(+1.56\) points (\(p = 0.00\))

⛓️ Proposed Solution: Decentralized Commit-Reveal Protocol

To eliminate post-hoc manipulation and reduce the auditing burden on independent researchers, the authors introduce a decentralized trust architecture utilizing Autonomous Economic Agents on an Ethereum-compatible ledger:

Phase 1 (Commit): Each judge records a one-way cryptographic hash of its evaluation score combined with a secret salt before the candidate models' identities are publicly revealed.
Phase 2 (Reveal): Candidate identities and raw scores are subsequently disclosed and cryptographically verified on-chain.

This mechanism successfully enforces a strict separation between blind evaluations and final score claims, creating a fully transparent and tamper-evident audit trail.

⛓️ Proposed Solution: Decentralized Commit-Reveal Protocol

To eliminate post-hoc manipulation and reduce the auditing burden on independent researchers, the authors introduce a decentralized trust architecture utilizing Autonomous Economic Agents on an Ethereum-compatible ledger:

Phase 1 (Commit): Each judge records a one-way cryptographic hash of its evaluation score combined with a secret salt before the candidate models' identities are publicly revealed.

Phase 2 (Reveal): Candidate identities and raw scores are subsequently disclosed and cryptographically verified on-chain.

This mechanism successfully enforces a strict separation between blind evaluations and final score claims, creating a fully transparent and tamper-evident audit trail.