跳转至

文章背景与核心概要

传统上，多语言推理模型的评估往往只关注其是否得出正确答案，而常常忽视它们在推理和回答过程中是否保持了预期的语言。这种忽视掩盖了随着任务难度增加而显现的关键多语言行为。

本文中，作者利用 PolyMath 基准测试（涵盖八种语言和四个难度级别），对推理模型中的任务难度、任务准确率、思考语言一致性（TC）以及回答语言一致性（AC）进行了深入研究。他们的研究结果表明，多语言性能不能仅通过准确率来准确描述；稳健的评估必须同时考虑任务准确率、语言一致性和任务难度。

推理型大语言模型中的隐藏语言一致性现象

作者： Muhammad Ali Shafique, Kelly Marchisio
提交时间： 2026年8月9日
主要学科： 计算与语言 (cs.CL)
次要学科： 人工智能 (cs.AI)
arXiv ID： 2608.08447

📌 摘要

Multilingual reasoning models are traditionally evaluated solely on whether they arrive at the correct answer, often overlooking whether they maintain the intended language during their reasoning and response processes. This oversight conceals critical multilingual behaviors that surface as task difficulty increases.

In this paper, the authors investigate task difficulty, task accuracy, thinking-language consistency (TC), and answer-language consistency (AC) across reasoning models using the PolyMath benchmark—spanning eight languages and four difficulty levels. Their findings reveal that multilingual performance cannot be accurately characterized by accuracy alone; robust evaluation must simultaneously account for task accuracy, language consistency, and task difficulty.

多语言推理模型的评估传统上仅根据它们是否得出正确答案来进行，往往忽略了它们在推理和回答过程中是否保持了预期的语言。这种疏忽掩盖了随着任务难度增加而显现的关键多语言行为。

在本文中，作者使用 PolyMath 基准测试——跨越八种语言和四个难度级别——对推理模型中的任务难度、任务准确率、思考语言一致性（TC）和回答语言一致性（AC）进行了研究。他们的发现表明，仅凭准确率无法准确描述多语言性能；稳健的评估必须同时兼顾任务准确率、语言一致性和任务难度。

🔍 核心发现

  1. Difficulty-Dependent Consistency Behaviors: Language consistency exhibits four distinct behaviors as difficulty scales:
  2. Output-language consistency remains aligned with the input.
  3. Output-language consistency remains misaligned.
  4. Consistency degrades gradually.
  5. Consistency collapses abruptly.
  1. 依赖于难度的的一致性行为： 随着难度升级，语言一致性表现出四种截然不同的行为：
  2. 输出语言一致性与输入保持一致。
  3. 输出语言一致性保持不对齐状态。
  4. 一致性逐渐退化。
  5. 一致性发生突发性崩塌。
  1. The Language Consistency Breakdown Effect: The authors identify a breakdown effect where increased task difficulty triggers a sudden drop in output-language consistency. This phenomenon is particularly prominent in less strongly represented and non-Latin-script languages.
  1. 语言一致性崩溃效应： 作者发现了一种崩溃效应，即任务难度的增加会引发输出语言一致性的骤降。这一现象在代表性较弱和非拉丁字母的语言中尤为突出。
  1. Paradoxical Accuracy Preservation: Because of the breakdown effect, a model's accuracy can sometimes be preserved—or even improve—at harder difficulty levels as it defaults back to its internal dominant language.
  1. 矛盾的准确率保持： 由于崩溃效应，当模型回退到其内部的主导语言时，其准确率有时在更高难度级别下得以保持——甚至有所提升。
  1. Impact of Quantization: Quantization can either improve or degrade output-language consistency independently of its effect on raw accuracy. Notably, methods like GPTQ and AWQ frequently outperform AutoRound under tolerance-based voting (with \(\epsilon = 1.0\)).
  1. 量化的影响： 量化既可以改善也可以恶化输出语言一致性，这与其对原始准确率的影响相对独立。值得注意的是，在基于容忍度的投票（其中 \(\epsilon = 1.0\)）下，GPTQ 和 AWQ 等方法经常优于 AutoRound。

🔗 访问与资源

license icon

license icon