跳转至

文章背景与核心概要

传统的基准测试通常仅使用单一的固定措辞来评估大语言模型（LLM）的性能，并默认这种单一表述能够代表问题的全部语义空间。然而，这种假设往往忽略了模型对措辞变化的敏感性。本文引入了 BenchDrift 框架，通过在语言、指代、语用和结构四个维度上生成保持语义不变的问题变体，系统地量化了模型性能的“漂移”现象。

研究发现，措辞敏感性并非随着模型能力的提升而消失，反而呈现出一种“符号反转”的特性：较弱的模型往往能从重述中获益，而顶尖模型却因措辞变化遭受显著的性能损失，这表明当前表现最好的模型在很大程度上依赖于特定的问题表述。此外，研究指出这种脆弱性更多源于措辞本身而非模型架构，不同模型在哪些重述会导致正确率下降的问题上表现出高度的一致性。

措辞效应：量化大语言模型基准测试中的双向漂移

The Wording Effect: Quantifying Two-Way Drift in LLM Benchmark Performance

摘要

Summary

Traditional Large Language Model (LLM) benchmarks evaluate performance using a single fixed phrasing for each problem, assuming it represents the entire problem space. This paper introduces BenchDrift, a framework that evaluates performance variance ("drift") by generating meaning-preserving variations of benchmark problems across linguistic, referential, pragmatic, and structural axes. Evaluating eight models across three benchmarks (GSM8K, MMLU, and MATH-Hard), the study reveals that rephrasing frequently flips correct answers into failures and vice versa. Key findings include: 1. Phrasing sensitivity scales with model capability: Rather than fading as models improve, sensitivity shifts in sign. Weak models net-gain from rephrasing, while strong models suffer significant net-losses, meaning top-performing models are often the most fragilely dependent on specific phrasings. 2. Fragility is a property of the wording, not the model: Different models largely agree on which specific rephrasings incur the heaviest costs in correct answers, regardless of their overall drift magnitude.

论文元数据

Paper Metadata

  • arXiv ID: arXiv:2608.11694 [cs.CL]
  • Submission Date: August 12, 2026
  • Authors: Shailja Thakur, Sungeun An, Chad DeLuca, Hima Patel
  • Primary Subject: Computation and Language (cs.CL)
  • Secondary Subject: Artificial Intelligence (cs.AI)
  • Code & Data: GitHub Repository (BenchDrift)

摘要

Abstract

A benchmark score comes from a single phrasing of each problem. That single phrasing is treated as if it stood for the whole space of ways the same problem could be asked, but it does not. We show that rephrasing a problem while keeping its meaning and answer fixed routinely flips a model's answer in both directions, so some failures become successes and some successes become failures. We call this drift.

BenchDrift generates meaning-preserving variations of benchmark problems along four axes, namely linguistic, referential, pragmatic, and structural, and measures how often, and why, correctness flips under each. Across eight models and three benchmarks (GSM8K, MMLU, MATH-Hard), we observe that drift is large in both directions.

Two findings stand out. * First, phrasing sensitivity does not fade as models get better. Instead, it changes sign. Weak models gain more from rephrasing than they lose, while strong models lose far more than they gain. We find that the best models on a benchmark are therefore the ones whose scores depend most on the wording they happened to be given. * Second, the models largely agree on which rephrasings cost the most correct answers even though they differ in how much they drift, so fragility belongs to the rephrasing and not to the model. Furthermore, rephrasing breaks answers a model was confident about, whether the problem is made shorter or longer.

全文及参考链接

Full-Text & Reference Links

license icon