跳转至

听见却未听取：音频-语言模型中的副语言信息编码与丢失

文章背景与核心概要

本文探讨了现代音频-语言模型（ALMs）在处理语音时，是否真正捕捉到了“如何说”（副语言线索、语调和说话风格）以及“说什么”（语义内容）。随着多模态大语言模型的快速发展，语音输入已成为人机交互的重要方式，但模型在处理情感、语气等非语义信息时的内部机制仍缺乏深入研究。

研究人员利用具有受控说话风格的 Expresso 数据集，对四个开源模型（Whisper-large-v2、Qwen2-Audio-7B Instruct、Qwen2.5-Omni-7B 和 Chroma-4B）进行了机制分析。研究发现，所有评估的模型在其编码器的后部层中均强烈编码了说话风格，但这些丰富的副语言信息在到达最终输出之前却持续退化。此外，研究引入了“内容-韵律泄漏度量”（content prosody leakage metric）来量化不同模型的输出行为，揭示了当前音频-语言模型在处理和应用副语言信号方面存在的根本性局限。

This paper investigates whether modern Audio-Language Models (ALMs) truly capture how something is spoken (paralinguistic cues, tone, and speaking style) alongside what is spoken.

Using the Expresso dataset with controlled speaking styles, the authors perform a mechanistic analysis of four open-source models: * Whisper-large-v2 * Qwen2-Audio-7B Instruct * Qwen2.5-Omni-7B * Chroma-4B

Key Findings

  1. Encoding vs. Utilization Gap: All evaluated models strongly encode speaking styles in the late encoder layers (the top third of the audio encoder). However, this rich paralinguistic information is consistently degraded before it reaches the final output.
  2. Role of Projectors and Decoders: The representation geometry is reshaped by the projector without necessarily dropping information, whereas decoders vary wildly in style preservation based on their architecture and training objectives.
  3. Model Output Behaviors: At the output stage, models split into two distinct operational paradigms:
  4. Content-Driven Models: Predictions rely overwhelmingly on text and semantic content.
  5. Acoustic-Driven Models: Predictions dynamically vary depending on the underlying speaking style.

Ultimately, the study introduces a content prosody leakage metric to quantify these behaviors, exposing a fundamental limitation in current audio-language models regarding how they process and apply paralinguistic signals.

文档元数据

Document Metadata

作者

  • Bhuvan Koduru
  • Dareen Safar B Alharthi
  • Rita Singh
  • Bhiksha Raj

Authors

  • Bhuvan Koduru
  • Dareen Safar B Alharthi
  • Rita Singh
  • Bhiksha Raj

访问链接