跳转至

零差距不等同于恢复：基准测试污染的分层逐题概率评估与逐步缓解策略

文章背景与核心概要

随着大语言模型规模的不断扩张，公开的基准测试数据不可避免地泄露到了预训练语料库中，导致模型通过死记硬背而非真正的推理能力，人为地推高了评估分数。尽管现有的“污染缓解评估”试图在解码阶段进行干预以抑制这种死记硬背，从而恢复模型的真实能力，但目前主流的指标——G-AP（聚合性能差距）存在根本性缺陷，主要表现为依赖离散读数、抑制误差相互抵消以及采用统一权重。

为了解决这一问题，本文作者引入了 SA-PPG（逐题概率差距分层聚合）指标，以实现更严格的评估；同时提出了 RailCap，一种创新的逐步缓解策略，能够在文本生成过程中对污染进行判定并动态设限。在各种受污染的模型和基准测试中，SA-PPG 证明了以往的缓解方法显著高估了真实能力的恢复程度，而 RailCap 则实现了最低的 SA-PPG，展现出更卓越的防污染效果。

核心贡献与方法论 (Key Contributions & Methodology)

现有指标（G-AP）的缺陷

  • 离散读数： 二元的正确/错误评估无法捕捉细致的逐题表现。
  • 误差抵消： 先求平均再求差值，导致过度抑制与抑制不足的误差相互抵消。
  • 统一权重： 对所有问题一视同仁，促使策略将求解概率推向清洁模型的高频值。

Flaws in Existing Metrics (G-AP)

  • Discrete Readouts: Binary correct/incorrect evaluations fail to capture nuanced per-question performance.
  • Error Cancellation: Averaging before differencing allows over-suppression and under-suppression to cancel each other out.
  • Uniform Weighting: Treating all questions equally encourages strategies to push solve probabilities toward the clean model's high-frequency values.

提出的评估方法：SA-PPG

SA-PPG（Stratified Aggregate of Per-question Probability Gaps，逐题概率差距分层聚合）通过以下方式解决了这些局限性： 1. 通过严格的采样来估计每个问题的求解概率。 2. 在逐题的基础上计算与清洁模型的概率差。 3. 在由清洁模型基准求解概率定义的特定层（组）内聚合这些差值。

Proposed Evaluation: SA-PPG

SA-PPG (Stratified Aggregate of Per-question Probability Gaps) solves these limitations by: 1. Estimating each question's solve probability via rigorous sampling. 2. Computing the probability difference against a clean model on a per-question basis. 3. Aggregating these differences within specific strata (groups) defined by the clean model's baseline solve probability.

提出的缓解策略：RailCap

传统策略在对污染进行操作之前先估计其发生的位置，这导致它们受制于初始估计的准确性。RailCap 改变了这一范式，它在生成过程中动态评估污染： * 每当生成的样本回落到贪婪轨迹（greedy trajectory）时，后续的轨迹 Token 就会被严格限制在候选次优解（runner-up）范围内。 * 这样会迭代累积抑制效果，直到响应分布变得足够分散。

Proposed Mitigation: RailCap

Traditional strategies estimate where contamination occurs before operating on it, making them bottlenecked by the accuracy of that initial estimate. RailCap changes this paradigm by evaluating contamination dynamically during generation: * Whenever a generated sample falls back onto the greedy trajectory, the subsequent trajectory token is strictly capped to the runner-up. * This accumulates suppression iteratively until the response distribution becomes sufficiently dispersed.

2. 主要发现 (Key Findings)

  • 高估了恢复效果： 通过在多个受污染模型和标准基准上进行 SA-PPG 评估，研究表明以往的缓解方法大大高估了它们恢复模型真实能力的有效性。
  • 卓越的性能： RailCap 始终优于先前的方法，实现了最低的 SA-PPG，并为抵御基准测试污染提供了更可靠的保障。

2. Key Findings

  • Overestimated Restoration: Through SA-PPG evaluation across multiple contaminated models and standard benchmarks, prior mitigation methods are revealed to have substantially overestimated how effectively they restore genuine model capabilities.
  • Superior Performance: RailCap consistently outperforms prior approaches, achieving the lowest SA-PPG and offering a more reliable safeguard against benchmark contamination.