文章背景与核心概要
位置编码（PE）对于使具有置换不变性的 Transformer 理解序列顺序至关重要，然而位置信息被处理和存储的内部机制在很大程度上仍然不够清晰。RoPE 等现代技术在处理长文本理解和检索任务时依然面临挑战。
为了深入研究这一问题，作者基于一个观察：在训练好的 Transformer 中，位置信号和语义信号占据了几乎正交的子空间。他们对编码器 Transformer 进行了修改，使其能够处理三个显式解耦的数据流：1. 语义流（限制于掩码语言建模 [MLM] 目标）；2. 绝对位置（AP）流；3. 相对位置（RP）流。
研究发现，孤立的 AP 子空间会自发崩溃为捕捉文档整体结构的低频二维流形；注意力机制的头自然分化为结构导向和语义导向两组；标准编码器则存在宏观结构保留不足的局限。通过这种解耦方法显式保留位置编码，成功提升了语言表征能力，在 Flash-Holmes 探查基准的 65 种语言现象中，有 49 种获得了更高分数。
Give it Space! Explicit Disentangling of Positional and Semantic Representations in Encoders
arXiv ID: arXiv:2605.30022 [cs.CL]
Authors: Pierre-Antoine Lequeu, Camille Barboule, Benjamin Piwowarski
Submission History: Submitted on 28 May 2026; Last revised on 2 September 2026 (v2).
arXiv ID: arXiv:2605.30022 [cs.CL]
Authors: Pierre-Antoine Lequeu, Camille Barboule, Benjamin Piwowarski
Submission History: Submitted on 28 May 2026; Last revised on 2 September 2026 (v2).
📌 Summary
Positional encodings (PE) are fundamental for enabling permutation-invariant Transformers to understand sequence order, yet the internal mechanisms of how positional information is processed and stored remain largely unclear. Modern techniques like RoPE continue to struggle with long-context understanding and retrieval tasks.
To investigate this, the authors build upon the observation that positional and semantic signals occupy nearly orthogonal subspaces in trained Transformers. They modify an encoder Transformer to process three explicitly disentangled streams: 1. Semantic stream (restricted to the masked-language-modeling [MLM] objective) 2. Absolute Positional (AP) stream 3. Relative Positional (RP) stream
Key Takeaways
- Low-Frequency Manifold Collapse: The isolated AP subspace spontaneously collapses into a low-frequency two-dimensional manifold that captures the overall structure of the document.
- Attention Head Specialization: Attention heads naturally specialize into structure-oriented and semantic-oriented groups, with relative positional (RP) streams exclusively supporting the latter.
- Limitations of Standard Encoders: Standard positional encodings fail to robustly retain macroscopic structure—RoPE and RP encode it only weakly, while entangled AP loses this structure in final layers under MLM pressure.
- Improved Linguistic Representation: Preserving positional encoding via this disentangled approach successfully improves linguistic representation, scoring higher on 49 out of 65 linguistic phenomena in the Flash-Holmes probing benchmark.
📌 Summary
Positional encodings (PE) are fundamental for enabling permutation-invariant Transformers to understand sequence order, yet the internal mechanisms of how positional information is processed and stored remain largely unclear. Modern techniques like RoPE continue to struggle with long-context understanding and retrieval tasks.
To investigate this, the authors build upon the observation that positional and semantic signals occupy nearly orthogonal subspaces in trained Transformers. They modify an encoder Transformer to process three explicitly disentangled streams: 1. Semantic stream (restricted to the masked-language-modeling [MLM] objective) 2. Absolute Positional (AP) stream 3. Relative Positional (RP) stream
Key Takeaways
- Low-Frequency Manifold Collapse: The isolated AP subspace spontaneously collapses into a low-frequency two-dimensional manifold that captures the overall structure of the document.
- Attention Head Specialization: Attention heads naturally specialize into structure-oriented and semantic-oriented groups, with relative positional (RP) streams exclusively supporting the latter.
- Limitations of Standard Encoders: Standard positional encodings fail to robustly retain macroscopic structure—RoPE and RP encode it only weakly, while entangled AP loses this structure in final layers under MLM pressure.
- Improved Linguistic Representation: Preserving positional encoding via this disentangled approach successfully improves linguistic representation, scoring higher on 49 out of 65 linguistic phenomena in the Flash-Holmes probing benchmark.
📋 Additional Metadata
| Category | Details |
|---|---|
| Primary Subject | Computation and Language (cs.CL) |
| Secondary Subjects | Artificial Intelligence (cs.AI) |
| MSC Classes | 68T50 |
| ACM Classes | I.2.7 |
| DOI | 10.48550/arXiv.2605.30022 |
| License | Creative Commons Attribution 4.0 |
📋 Additional Metadata
Category Details Primary Subject Computation and Language ( cs.CL)Secondary Subjects Artificial Intelligence ( cs.AI)MSC Classes 68T50 ACM Classes I.2.7 DOI 10.48550/arXiv.2605.30022 License Creative Commons Attribution 4.0
🔗 Access Links
🔗 Access Links