跳转至

混合专家（MoE）模块包含强有力的幻觉检测信号

文章背景与核心概要

大语言模型（LLM）虽然得到了广泛应用，但仍然饱受生成貌似合理实则错误的信息（即“幻觉”）这一根本问题的困扰。传统的检测方法通常停留在答案或句子级别，而词元（token）级别的检测对于定位幻觉内容和实现细粒度干预至关重要。本文介绍了 InnerExpert，这是一种创新方法，它利用内部混合专家（MoE）信号（如路由器熵、专家分歧和使用模式）来实现逐词元的幻觉检测。通过将路由级指标与Transformer特征相结合，InnerExpert在单次前向传播中即可在多个基准测试中实现业界领先的性能。

Executive Summary

大语言模型（LLM）经常会生成看似合理实则错误的信息，这种现象被称为幻觉。虽然传统的检测方法通常在答案或句子级别进行，但逐词元（per-token）检测对于定位幻觉内容至关重要。本文推出了 InnerExpert，这是一种新颖的方法，它利用内部混合专家（MoE）信号（例如路由器熵、专家分歧和使用模式）进行逐词元幻觉检测。通过将路由级指标与Transformer特征相结合，InnerExpert在单次前向传播中实现了多个基准测试下的最先进性能。

Executive Summary

Large Language Models (LLMs) frequently suffer from generating plausible yet false information, a phenomenon known as hallucination. While traditional detection methods typically operate at the answer or sentence level, per-token detection is crucial for localizing hallucinated content. This paper introduces InnerExpert, a novel approach that leverages internal Mixture-of-Experts (MoE) signals—such as router entropy, expert disagreement, and usage patterns—for per-token hallucination detection. By combining routing-level metrics with transformer features, InnerExpert achieves state-of-the-art performance across multiple benchmarks in a single forward pass.

Paper Overview

  • 标题： 混合专家（MoE）模块包含强有力的幻觉检测信号
  • 作者： Joao Fonseca, Rodrigo Rodrigues, Paolo Romano
  • 主要学科： 计算机科学 > 人工智能 (cs.AI)
  • 次要学科： 机器学习 (cs.LG)
  • 提交时间： 2026年8月18日
  • arXiv 标识符： arXiv:2608.17687 [cs.AI]
  • DOI： 10.48550/arXiv.2608.17687

Paper Overview

  • Title: Mixture-of-Expert Blocks Contain Strong Hallucination Detection Signals
  • Authors: Joao Fonseca, Rodrigo Rodrigues, Paolo Romano
  • Primary Subject: Computer Science > Artificial Intelligence (cs.AI)
  • Secondary Subject: Machine Learning (cs.LG)
  • Submitted on: August 18, 2026
  • arXiv Identifier: arXiv:2608.17687 [cs.AI]
  • DOI: 10.48550/arXiv.2608.17687

Abstract

尽管大语言模型（LLM）被广泛使用，但它们仍然受限于一个根本问题：生成貌似合理但错误的内容，即幻觉。大多数现有的检测方法都在答案或句子级别运行，然而逐词元检测对于定位幻觉片段和实现细粒度干预至关重要。

在本文中，作者探讨了使用混合专家（MoE）范式来填补这一空白。在MoE架构中，单次前向传播通过路由机制激活稀疏的专家子集（即每层包含不同的前馈网络），从而产生内部信号（例如路由器熵、专家分歧和专家使用模式），这些信号在密集（dense）架构中是不可用的，且此前未被开发用于幻觉检测。

为此，作者推出了 InnerExpert，这是第一种利用这些MoE特定信号进行逐词元幻觉检测的方法。InnerExpert将路由级信号和标准Transformer信号组合成紧凑的逐词元特征向量，并由轻量级检测器对其进行分类。该检测器通过由LLM作为裁判（LLM-as-a-judge）管线产生的标签进行训练，从而无需手动标注即可实现模型的持续更新。

结果表明，InnerExpert在五个数据集和两个MoE架构上均优于现有方法，最高实现了： * 0.91 的答案级 AUROC * 0.76 的词元级 AUROC

……所有这一切仅需单次前向传播即可完成。

Abstract

Despite their widespread use, Large Language Models (LLMs) remain limited by a fundamental problem: the generation of plausible but false content, known as hallucinations. Most existing detection methods operate at the answer or sentence level, yet per-token detection is essential for localizing hallucinated spans and enabling fine-grained interventions.

In this paper, the authors explore the use of the Mixture-of-Experts (MoE) paradigm to address this gap. In MoE architectures, a single forward pass activates a sparse subset of experts (i.e., distinct feedforward networks per layer) via a routing mechanism, producing internal signals (e.g., router entropy, expert disagreement, and expert usage patterns) that are unavailable in dense architectures and have not been previously exploited for hallucination detection.

To this end, the authors introduce InnerExpert, the first method to leverage these MoE-specific signals for per-token hallucination detection. InnerExpert combines routing-level and standard transformer signals into compact per-token feature vectors, classified by a lightweight detector trained on labels produced by an LLM-as-a-judge pipeline, which enables continuous model updates without manual annotation.

Results demonstrate that InnerExpert outperforms existing methods across five datasets and two MoE architectures, achieving up to: * 0.91 answer-level AUROC * 0.76 token-level AUROC

...all while requiring only a single forward pass.

Key Contributions & Features

  • MoE信号的开发： 利用混合专家层特有的内部指标——如路由器熵、专家使用模式和专家分歧，这些指标在传统的密集模型中是不存在的。
  • InnerExpert框架： 首个专门设计用于将路由级和Transformer级指标融合为紧凑逐词元特征向量的框架。
  • 自动化标注管线： 采用“LLM作为裁判”的方法动态生成训练标签，促进了持续更新，省去了昂贵的手动标注。
  • 高性能与高效率： 采用计算高效的单次前向传播设计，在五个不同的数据集和两个MoE架构上超越了基线方法。

Key Contributions & Features

  • Exploitation of MoE Signals: Utilizes internal metrics unique to Mixture-of-Expert layers—such as router entropy, expert usage patterns, and expert disagreement—which are absent in traditional dense models.
  • InnerExpert Framework: The first framework designed specifically to fuse routing-level and transformer-level metrics into compact per-token feature vectors.
  • Automated Labeling Pipeline: Employs an LLM-as-a-judge approach to generate training labels dynamically, facilitating continuous updates without costly manual annotation.
  • High Performance & Efficiency: Surpasses baseline approaches over five distinct datasets and two MoE architectures using a computationally efficient single-forward-pass design.

Access & Resources

Access & Resources