跳转至

针对一个模型封闭，对另一个模型开放：法律多项选择基准测试中的“纯选项可解性”

文章背景与核心概要

多项选择基准测试（Multiple-choice benchmarks）通常根据模型是否选出正确选项来进行评分，而忽视了模型在得出答案时是否真的需要阅读题目。本文深入探讨了“纯选项可解性”（option-only solvability，即在不提供题干的情况下仅凭选项回答问题），并基于乌克兰高等法官资格委员会发布的包含11,990道四选一题目的数据集 UA-JudgeExam 展开了全面研究。

研究揭示了当前基准测试验证中的关键漏洞：首先是“盲猜泄露”（Blind Leaks），某些模型（如 Claude Haiku 4.5）在所有选项顺序下都能盲猜答对相当比例的题目，这源于特定的格式脆弱性，而非背诵了法律条文；其次是“模型特定门控”（Model-Specific Gating），即基于某个模型过滤掉脆弱题目并不能可靠地转移到其他模型上（例如 GPT-5.6 仍然能盲猜解开许多被“过滤”掉的题目）；最后，如果不控制位置偏差（如默认选择“A”），模型排行榜可能会产生极大误导。该作者已在 Hugging Face 上发布了完整的语料库、模型预测结果及评估工具。

📌 Summary

Multiple-choice benchmarks often evaluate models based on whether they select the correct option, rather than whether they actually required the question to arrive at the answer. This paper investigates option-only solvability (answering questions correctly without being shown the question stem) using UA-JudgeExam, a dataset of 11,990 four-option items published by Ukraine's Higher Qualification Commission of Judges.

The study reveals critical flaws in benchmark validation: * Blind Leaks: Certain models (e.g., Claude Haiku 4.5) can answer a significant portion of questions blindly across all option orders, driven by specific format vulnerabilities rather than verbatim text quotation from legal codes. * Model-Specific Gating: Filtering out vulnerable items based on one model does not reliably transfer to other models (e.g., GPT-5.6 can still solve many of the "filtered" items blindly). * Positional Biases: Without controlling for answer-position habits (such as defaulting to option "A"), model leaderboards can be severely misleading. * Format Dependency: Item formats dictate whether option-only solvability arises, whereas model capabilities dictate how much advantage can be extracted from those vulnerabilities.

The author has released the full corpus, model predictions, and evaluation harness on Hugging Face.

多项选择基准测试通常根据模型是否选择了正确选项来评分，而不是考察它是否真的需要题目。衡量这种差距需要谨慎：只要正确答案位于 A，一个对大多数题目都回答 A 的模型其得分就会高于随机概率；而当答案不在 A 时，则会被误读为模型具备识别能力。我们在 UA-JudgeExam 上对此进行了测量：这是一个包含 11,990 道四选项题目及官方答案的白皮书数据集，由乌克兰高等法官资格委员会发布。

Multiple-choice benchmarks are graded on whether a model picks the right option, not on whether it needed the question. Measuring that gap takes care: a model answering A to most items scores above chance wherever the key sits at A, and reads as recognition when it is not. We measure it on UA-JudgeExam: 11,990 four-option items with official keys, published by Ukraine's Higher Qualification Commission of Judges.

在不给出题干、只展示选项的情况下，Claude Haiku 4.5 获得高出随机概率 0.383 的分数，且这种信息泄露高度集中：有 11.8% 的题目在所有八种选项排列顺序下均可被盲猜答对，而随机预期的数量仅为 0.2 道。这并非由于背诵原文：通过检索 280,059 版乌克兰立法文本，重合度仅为 0.128。过滤掉这些题目后剩余 8,128 道题，过滤模型本身在这部分题目上的得分降至 0.204，而未参与筛选过程的 GPT-5.6 在隐藏题干的情况下依然能盲猜答对其中 0.515 的题目。在对整个数据集测试的 12 个独立模型进行评分并扣除各自的答案位置偏好习惯后，仅有两个模型保持正向超额：GPT-5.6 达 +0.265，Sonnet 4.6 达 +0.081。如果不扣除位置偏好，排行榜将被严重误导：Llama 3.1 8B 的盲猜得分为 0.292，高居除上述两个模型之外的所有模型之上，其纯粹原因在于它对 92% 的题目都选了 A。

Shown the options and no question, Claude Haiku 4.5 scores 0.383 against chance, and the leak is concentrated: 11.8% of items are answered blind on all eight option orders, against 0.2 items expected by chance. It is not quotation: search over 280,059 editions of Ukrainian legislation recovers 0.128. Gating those out retains 8,128 items, on which the gating model itself now scores 0.204, and GPT-5.6, which took no part in the selection, still answers 0.515 of them with the question hidden. Scoring twelve held-out models on the whole set and subtracting each one's answer-position habit, only two keep an excess: GPT-5.6 at +0.265, Sonnet 4.6 at +0.081. Without it the ranking misleads: Llama 3.1 8B scores 0.292 blind, above every model but those two, purely by answering A to 92% of items.

这种门控机制确实筛选出了一些真实存在的问题：在其拒绝的题目上，12 个模型中有 11 个的得分在 0.518-0.789 之间，每个区间的得分都明显高于同一模型在保留题目上的得分。但这种信号属于特定模型，基于它进行的过滤无法向上迁移。在 400 个样本的子集上，这些现象均不可见，其中 9 个模型被判定为“在统计学上等同于随机猜测”。相比之下，重写干扰项（distractors）则矫枉过正至 0.168，低于随机水平且同样极易被利用。在 LEXam 上进行同样的探测则呈现随机结果：那里的每一个选项都直接指向题干，且没有任何选项长度超过 33 个字符。题目格式决定了纯选项可解性问题是否会出现，而模型能力则决定了能从这些漏洞中提取出多少优势。我们现已公开语料库、预测结果及评估工具。

The gate does select something real: on the items it rejected, eleven of twelve models score 0.518-0.789, every interval clear of what the same model scores on the items it kept. But that signal is one model's, and filtering on it does not transfer upward. Neither is visible on a 400-item sample, where nine models read as "statistically at chance". Rewriting distractors instead overshoots to 0.168, below chance and as exploitable. The same probe on LEXam returns chance: every option there points into the stem, none longer than 33 characters. Item format decides whether the problem can arise; capability decides how much is extracted. We release the corpus, the predictions and the harness.

license icon (View license details via Creative Commons Attribution 4.0)