跳转至

文章背景与核心概要

从非侵入式大脑记录（如 EEG 和 MEG）中直接解码感知语音，因其在现实世界中的巨大应用潜力而受到广泛关注。然而，现有模型普遍存在跨受试者泛化能力差的问题，且缺乏在不同个体之间提取一致性信息的机制，从而导致训练成本高昂且性能欠佳。

为了克服这些障碍，研究人员提出了跨受试者感知语音解码（CPSD）框架。该框架包含三大核心创新：1）源模型预训练，利用对比学习从多个源受试者中提取共享表征；2）个性化专用微调，利用从源模型中提取的一致性组件来初始化目标受试者模型，随后使用目标数据进行微调；3）基于位置编码的空间注意力（PESA）模块，将 MEG/EEG 数据重新映射到标准化的参考空间中，以最大化跨受试者的一致性并简化训练流程。

该框架在三个多模态、多语言的感知语音数据集上进行了评估。实验结果表明，与现有基线相比，CPSD 框架显著提升了 Top-10 准确率：在 Armeni 2022 数据集上提升了 6.8%，在 PKUEEG 2025 数据集上提升了 15.4%，在 Broderick 2018 数据集上提升了 15.8%。

Cross-Subject Generalization in Decoding Perceived Speech from Non-Invasive Brain Recordings

Summary

Decoding perceived speech directly from non-invasive brain recordings (such as EEG and MEG) has gained immense popularity due to its potential real-world applications. However, existing models suffer from poor cross-subject generalizability and lack mechanisms to extract consistent information across different individuals, resulting in high training costs and suboptimal performance.

To overcome these hurdles, researchers propose the Cross-Subject Peressed Speech Decoding (CPSD) framework, which features: 1. Source Model Pre-training: Utilizing contrastive learning to extract shared representations from multiple source subjects. 2. Personal Specialization: Initializing target subject models using consistent components extracted from the source model, followed by fine-tuning with target data. 3. Positional Encoding-based Spatial Attention (PESA) Module: Remapping MEG/EEG data into a standardized reference space to maximize cross-subject consistency and streamline training.

Evaluated across three multi-modality and multi-language perceived speech datasets, the CPSD framework significantly outperforms existing baselines, achieving improvements in Top-10 accuracy of 6.8% (Armeni 2022), 15.4% (PKUEEG 2025), and 15.8% (Broderick 2018).

Paper Metadata

论文元数据

  • arXiv ID: arXiv:2608.22420 [cs.SD]
  • Primary Subject: Computer Science > Sound (cs.SD)
    • Primary Subject: Computer Science > Sound (cs.SD)
  • Secondary Subjects: Artificial Intelligence (cs.AI)
    • Secondary Subjects: Artificial Intelligence (cs.AI)
  • Submitted Date: August 23, 2026
    • Submitted Date: August 23, 2026
  • Submitted To: IEEE Transactions on Audio, Speech, and Language Processing (TASLP)
    • Submitted To: IEEE Transactions on Audio, Speech, and Language Processing (TASLP)
  • DOI: 10.48550/arXiv.2608.22420

Authors

作者团队

  • Aoke Zhang
    • Aoke Zhang
  • Bo Wang
    • Bo Wang
  • Xihong Wu
    • Xihong Wu
  • Heping Cheng
    • Heping Cheng
  • Jing Chen
    • Jing Chen

Abstract

摘要

从非侵入式脑电记录中解码感知语音近年来因其广泛的应用前景而备受关注。然而，现有方法在跨受试者解码方面面临巨大挑战，这主要是由于泛化能力有限以及缺乏用于提取受试者间一致性信息的显式机制。这些局限性导致训练成本高昂且解码性能不佳。

Decoding perceived speech from non-invasive brain recordings has garnered significant attention in recent years due to its wide range of potential applications. However, existing methods face considerable challenges in cross-subject decoding, primarily due to limited generalizability and the absence of explicit mechanisms for extracting subject-consistent information. These limitations result in high training costs and suboptimal decoding performance.

为了应对这些挑战，我们提出了一种创新的跨受试者感知语音解码（CPSD）框架，该框架包含两个训练阶段：源模型预训练和个性化专用微调。在源模型预训练阶段，采用对比学习来捕获多个源受试者之间的共享表征。随后，个性化专用阶段通过从源模型中提取一致性组件来初始化目标受试者模型，并使用目标受试者数据对其进行微调。

To address these challenges, we propose an innovative Cross-Subject Perceived Speech Decoding (CPSD) framework, which comprises two training stages: source model pre-training and personal specialization. In the source model pre-training stage, contrastive learning is employed to capture shared representations across multiple source subjects. Subsequently, personal specialization initializes the model for the target subject by extracting consistent components from the source model and fine-tuning it using target subject data.

此外，我们引入了基于位置编码的空间注意力（PESA）模块，将 MEG/EEG 数据重新映射到标准化的参考空间中，从而增强了跨受试者的一致性并简化了模型训练。我们在包含不同模态和语言的三个感知语音神经数据集上评估了所提出的 CPSD 框架。结果表明，在 Armeni 2022、PKUEEG 2025 和 Broderick 2018 数据集上，我们的框架在 Top-10 准确率上分别比基线方法高出 6.8%、15.4% 和 15.8% 以上。进一步的分析证实了所提方法的有效性、高效性和鲁棒性。

Additionally, we introduce the Positional Encoding-based Spatial Attention (PESA) module, which remaps MEG/EEG data into a standardized reference space, thereby enhancing cross-subject consistency and facilitating model training. We evaluate the proposed CPSD framework on three perceived speech neural datasets encompassing different modalities and languages. The results demonstrate that our framework outperforms baseline methods by more than 6.8%, 15.4%, and 15.8% in Top-10 accuracy on the Armeni 2022, PKUEEG 2025, and Broderick 2018 datasets, respectively. Further analyses confirm the effectiveness, efficiency, and robustness of the proposed approach.

Full-Text & Reference Links

全文与参考链接