跳转至

文章背景与核心概要

本文探讨了多语言大语言模型（LLM）在不同语言中解决推理任务时，究竟是利用了共享的内部特征，还是依赖于仅仅产生相似输出的特定语言计算。通过使用多语言小学数学（MGSM）数据集对四个家族的五个模型进行分析，作者引入了一种新颖的几何不变稀疏自编码器（GI-SAE）。该方法使用 InfoNCE 损失来鼓励编码器对相同的推理轨迹产生一致的激活，而与语言无关。研究得出结论：虽然 GI-SAE 成功提高了几何相似性，但这并不一致地转化为功能上的可互换性，这表明跨语言特征共享高度依赖于模型架构和层深度。

文章背景与核心概要

This paper investigates whether multilingual Large Language Models (LLMs) utilize shared internal features when solving reasoning tasks across different languages, or if they rely on language-specific computations that merely yield similar outputs. By analyzing five models across four families using the Multilingual Grade School Math (MGSM) dataset, the authors introduce a novel Geometry-Invariant Sparse Autoencoder (GI-SAE). This method uses an InfoNCE loss to encourage the encoder to produce consistent activations for identical reasoning traces regardless of language. The study concludes that while GI-SAE successfully increases geometric similarity, this does not consistently translate to functional interchangeability, revealing that cross-language feature sharing is highly dependent on both model architecture and layer depth.

通过几何不变稀疏自编码器发现大语言模型中的跨语言推理不变性

Discovering Cross-Language Reasoning Invariance in LLMs with Geometry-Invariant Sparse Autoencoders

作者： Igor Bogdanov, Changcheng Huang
日期： 2026年8月24日
会议： ICML 2026 机械可解释性研讨会
arXiv ID： 2608.23809

Authors: Igor Bogdanov, Changcheng Huang
Date: August 24, 2026
Venue: ICML 2026 Workshop on Mechanistic Interpretability
arXiv ID: 2608.23809

摘要

本文研究了多语言大语言模型（LLM）在不同语言中解决推理任务时，是否利用了共享的内部特征，还是依赖于仅仅产生相似输出的特定语言计算。通过使用多语言小学数学（MGSM）数据集对四个家族的五个模型进行分析，作者引入了一种新颖的几何不变稀疏自编码器（GI-SAE）。该方法使用 InfoNCE 损失来鼓励编码器对相同的推理轨迹产生一致的激活，而与语言无关。研究得出结论：虽然 GI-SAE 成功提高了几何相似性，但这并不一致地转化为功能上的可互换性，这表明跨语言特征共享高度依赖于模型架构和层深度。

Summary

This paper investigates whether multilingual Large Language Models (LLMs) utilize shared internal features when solving reasoning tasks across different languages, or if they rely on language-specific computations that merely yield similar outputs. By analyzing five models across four families using the Multilingual Grade School Math (MGSM) dataset, the authors introduce a novel Geometry-Invariant Sparse Autoencoder (GI-SAE). This method uses an InfoNCE loss to encourage the encoder to produce consistent activations for identical reasoning traces regardless of language. The study concludes that while GI-SAE successfully increases geometric similarity, this does not consistently translate to functional interchangeability, revealing that cross-language feature sharing is highly dependent on both model architecture and layer depth.

研究概述

多语言模型展现出了用多种不同语言（英语、德语、法语、西班牙语、俄语和中文）解决相同数学问题的卓越能力。本研究旨在揭示这一能力背后的“机械”现实：

  • 方法论： 作者在回放有效推理轨迹的同时，记录了多个层的内部模型表示。
  • 分析工具：
    • 中心核对齐（CKA）： 用于识别具有高跨语言对齐的层。
    • 几何不变稀疏自编码器（GI-SAE）： 一种新颖的架构，它通过信息噪声对比估计（InfoNCE）损失来补充标准的重建损失，以强制实现语言不变的特征表示。
  • 功能测试： 研究人员通过在前向传递过程中在不同语言之间交换这些特征，并测量模型输出中由此产生的 KL 散度，来测试这些特征的“可互换性”。

Research Overview

Multilingual models demonstrate a remarkable ability to solve identical mathematical problems in diverse languages (English, German, French, Spanish, Russian, and Chinese). This research seeks to uncover the "mechanistic" reality behind this capability:

  • Methodology: The authors recorded internal model representations at multiple layers while replaying valid reasoning traces.
  • Analysis Tools:
    • Centered Kernel Alignment (CKA): Used to identify layers with high cross-language alignment.
    • Geometry-Invariant Sparse Autoencoder (GI-SAE): A new architecture that supplements standard reconstruction loss with an Information Noise-Contrastive Estimation (InfoNCE) loss to force language-invariant feature representations.
  • Functional Testing: The researchers tested the "interchangeability" of these features by swapping them between languages during forward passes and measuring the resulting KL divergence in the model's output.

核心发现

  • 几何不变性与功能不变性： 尽管 GI-SAE 显著提高了 CKA 和 Jaccard 相似性得分，但更高的几何对齐并不能保证这些特征在功能上是可互换的。
  • 模型特定行为： 跨语言特征的存在及其效用因模型而异：
    • Qwen： 表现出强化的跨语言结构。
    • Gemma： 从诱导出的几何对齐中没有获得功能上的益处。
    • Llama & Phi： 表现出混合的、依赖于层的效果。
  • 结论： LLM 中的跨语言推理并非普遍现象；它与特定的架构选择和模型的内部表示深度紧密相连。

Key Findings

  • Geometric vs. Functional Invariance: While GI-SAE significantly improves CKA and Jaccard similarity scores, higher geometric alignment does not guarantee that the features are functionally interchangeable.
  • Model-Specific Behavior: The presence and utility of cross-language features vary significantly by model:
    • Qwen: Shows strengthened cross-language structure.
    • Gemma: Shows no functional benefit from the induced geometric alignment.
    • Llama & Phi: Exhibit mixed, layer-dependent effects.
  • Conclusion: Cross-language reasoning in LLMs is not a universal phenomenon; it is deeply tied to specific architectural choices and the internal representation depth of the model.

访问与资源

Access & Resources