跳转至

文章背景与核心概要

现代混合序列模型（Hybrid Sequence Models）在设计时往往无法满足前缀不变性（prefix invariance）——即位置 \(t\) 处的Token表征绝不能依赖于未来输入这一基本要求。虽然仅依赖注意力掩码检查（Attention-mask inspection）是当前该领域的默认验证方法，但本文证明了此类检查并不完整，因为因果关系本质上是一个图级别的属性。尽管掩码正确，信息仍可能通过扫描（scans）、聚合（aggregations）或归一化层发生泄漏。

为了解决这一问题，作者引入了一种轻量级、无需梯度的审计方法，仅需两次前向传播即可生成精确的逐层得分。在针对八个检查点的192次注入故障试验中，传统的掩码检查未能检测出哪怕一次泄漏（0/192），而提出的审计方法成功将所有192/192的故障定位到了准确的层。此外，通过对Transformer中分块扫描（chunked-scan）代码进行静态与动态分析，研究人员在 Zamba2 和 Nemotron-H 中发现了此前隐藏的缺陷（块间轴错误），并随后使用参考实现对其进行了修复。

The Mask Is Not the Model: Auditing Prefix Invariance in Attention, State-Space, and Hybrid Sequence Models

arXiv: 2608.22876 [cs.LG]
Authors: Taebong Kim, Youngsik Hong, Minsik Kim, Sunyoung Choi, Jaewon Jang, Minseo Kim
Submitted: August 24, 2026 (Last revised: August 25, 2026)

arXiv: 2608.22876 [cs.LG]
Authors: Taebong Kim, Youngsik Hong, Minsik Kim, Sunyoung Choi, Jaewon Jang, Minseo Kim
Submitted: August 24, 2026 (Last revised: August 25, 2026)

📌 Summary

📌 Summary

现代混合序列模型经常无法满足前缀不变性——即位置 \(t\) 处的Token表征绝不能依赖于未来输入的根本要求。虽然纯粹依赖注意力掩码检查是该领域的默认验证方法，但本文证明此类检查是不完整的，因为因果关系最终是一个图级别的属性。尽管掩码正确，信息仍然可以通过扫描、聚合或归一化层发生泄漏。

Modern hybrid sequence models often fail to satisfy prefix invariance—the fundamental requirement that token representations at position \(t\) must not depend on future inputs. While relying purely on attention-mask inspection is the field's default validation method, this paper demonstrates that such checks are incomplete because causality is ultimately a graph-level property. Information leaks can still occur through scans, aggregations, or normalization layers despite correct masks.

为了解决这个问题，作者引入了一种轻量级、无需梯度的审计方法，只需两次前向传播即可生成精确的逐层得分。在跨8个检查点的192次注入故障试验中，传统掩码检查未能检测出哪怕一次泄漏（0/192），而所提出的审计成功将所有192/192次故障定位到其精确的层。此外，对Transformer中分块扫描代码进行静态和动态分析，发现了 Zamba2 和 Nemotron-H 中先前隐藏的缺陷（块间轴错误），这些缺陷随后通过参考实现得以修复。

To address this, the authors introduce a lightweight, gradient-free auditing method requiring only two forward passes to generate a precise per-layer score. Across 192 injected-fault trials on eight checkpoints, traditional mask inspection failed to detect a single leak (\(0/192\)), whereas the proposed audit successfully localized all \(192/192\) failures to their exact layers. Furthermore, applying static and dynamic analysis to chunked-scan code in transformers uncovered previously hidden defects in Zamba2 and Nemotron-H (an inter-chunk axis error), which were subsequently fixed using the reference implementation.

📝 Abstract

📝 Abstract

混合序列模型必须满足前缀不变性：位置 \(t\) 处的表征不能依赖于未来的输入，然而这一点很少得到验证。我们将前缀不变性形式化，并给出了一种轻量级的审计方法，仅需两次前向传播，无需训练或梯度，即可产生一个逐层得分，以定位因果关系断裂的位置。

注意力掩码检查作为该领域的默认检查手段是不完整的：因果关系是一个图级别的属性，尽管掩码正确，但泄漏仍可能通过扫描、聚合或归一化发生。在跨8个检查点的192次注入故障试验中，掩码检查没有检测到任何一次，而我们的审计将所有192/192次故障都定位到了确切的层。

对Transformer中分块扫描代码的静态/动态分析发现，Zamba2和Nemotron-H中存在相同的缺陷（块间轴错误），该错误已通过参考实现修复。该方法篇幅仅有一页，且可在几秒钟内运行。

Hybrid sequence models must satisfy prefix invariance: representations at position \(t\) must not depend on future inputs, yet this is rarely verified. We formalize prefix invariance and give a lightweight audit, two forward passes, no training or gradients, yielding a per-layer score localizing where causality breaks.

Attention-mask inspection, the field's default check, is incomplete: causality is a graph-level property, and leaks can occur via scans, aggregations, or normalization despite correct masks. Across 192 injected-fault trials on eight checkpoints, mask inspection detected none, while our audit localized all 192/192 to the exact layer.

Static/dynamic analysis of chunked-scan code in transformers found the same defect in Zamba2 and Nemotron-H, an inter-chunk axis error fixed via the reference implementation. The method fits on one page and runs in seconds.

🕒 Submission History

🕒 Submission History

  • [v1] 2026年8月24日 星期一 07:01:32 UTC (244 KB)
  • [v2] 2026年8月25日 星期二 04:49:06 UTC (244 KB)
  • [v1] Mon, 24 Aug 2026 07:01:32 UTC (244 KB)
  • [v2] Tue, 25 Aug 2026 04:49:06 UTC (244 KB)