跳转至

文章背景与核心概要

音频条件大语言模型（Audio-conditioned LLMs）在处理语音时，往往无法充分利用语调、情感和非语音声音等关键声学线索。本文旨在探究这是否是因为受ASR（语音识别）监督的前端在信息到达语言模型（LM）之前丢弃了这些信息。

通过在统一的 Qwen3.5-4B 音频-语言模型框架中评估各种前端（Whisper-Tiny、Whisper-Small、EnCodec、DAC-VAE 和 WavTokenizer），作者发现仅靠更换编码器并不能解决声学信息利用不足的问题。相反，线性探针（Linear probes）和几何分析表明，可区分的声学结构在语言模型的最后一层依然完全可恢复。因此，性能差距是由语言模型头部（LM head）内部的特定内容读取失败（content-specific readout failure）和读取对齐瓶颈（readout alignment bottlenecks）所导致的，而不是简单的编码器端信息丢失或通用的格式偏差。

文章背景与核心概要

Where Does the Sound Go? Tracing Acoustic Information Loss in Audio-Conditioned LLMs

arXiv ID: arXiv:2609.05871
Primary Subject: Sound (cs.SD)
Secondary Subjects: Artificial Intelligence (cs.AI)
Authors: Song-ha Jo, Sehyun Lee, Soyoon Kim, Jaesik Choi, Sanghyuk Choi
Submission Date: September 5, 2026
Conference: EMNLP 2026 Findings

📋 Summary

音频条件大语言模型经常未能充分利用语调、情感和非语音声音等关键声学线索。本文调查了ASR监督的前端是否应在信息到达语言模型（LM）之前丢弃该信息负责。通过在共享的 Qwen3.5-4B 音频-LM管道中评估各种前端（Whisper-Tiny、Whisper-Small、EnCodec、DAC-VAE和WavTokenizer），作者表明仅靠编码器替换并不能解决声学利用不足的问题。相反，线性探针和几何分析显示，可区分的声学结构在最后的LM层仍然完全可恢复。因此，性能差距是由LM头部内的特定内容读取失败和读取对齐瓶颈驱动的，而不是简单的编码器端信息丢失或通用格式偏差。

Audio-conditioned language models often underuse critical acoustic cues such as prosody, emotion, and non-speech sounds. This paper investigates whether ASR-supervised frontends are responsible for discarding this information before it reaches the language model (LM). By evaluating various frontends (Whisper-Tiny, Whisper-Small, EnCodec, DAC-VAE, and WavTokenizer) within a shared Qwen3.5-4B audio-LM pipeline, the authors show that encoder replacement alone does not resolve the acoustic underuse. Instead, linear probes and geometric analyses reveal that discriminative acoustic structure remains fully recoverable at the final LM layer. Consequently, the performance gap is driven by a content-specific readout failure and readout alignment bottlenecks within the LM head, rather than simple encoder-side information loss or generic format bias.

👥 Authors & Affiliations

  • Song-ha Jo
  • Sehyun Lee
  • Soyoon Kim
  • Jaesik Choi
  • Sanghyuk Choi

👥 Authors & Affiliations

  • Song-ha Jo
  • Sehyun Lee
  • Soyoon Kim
  • Jaesik Choi
  • Sanghyuk Choi

📄 Abstract

音频条件大语言模型往往未能充分利用语调、情感和非语音声音等声学线索，这就提出了一个问题：ASR监督的前端是否在信息到达LM之前就将其丢弃了。我们在ASR、情感识别和声音字幕生成任务中，通过在共享的Qwen3.5-4B音频-LM管道中比较Whisper-Tiny、Whisper-Small与EnCodec、DAC-VAE和WavTokenizer，来测试前端是否应为此负责。仅靠编码器替换并不能解决这种利用不足的问题：Whisper变体总体上仍然是最强的，包括在情感和环境声音字幕生成方面。

为了定位故障点，我们通过编码器、投影器、LM层和LM头追踪了任务相关的信息。线性探针和几何分析表明，即使多项选择问答（MCQA）的准确率落后探针准确率高达83个百分点，可区分的声学结构在最后的LM层仍然是可恢复的。由于回答格式和解码过程受到了控制，这种依赖于任务的差距指向了特定内容读取失败，而不是通用的格式偏差。LogitLens分析和有针对性的LM head干预支持了以下结论：声学信息利用不足不能仅仅用编码器端的信息丢失来解释，读取对齐可能是主要的瓶颈。

Audio-conditioned language models often underuse acoustic cues such as prosody, emotion, and non-speech sounds, raising the question of whether ASR-supervised frontends discard this information before it reaches the LM. We test whether the frontend is responsible by comparing Whisper-Tiny and Whisper-Small with EnCodec, DAC-VAE, and WavTokenizer in a shared Qwen3.5-4B audio-LM pipeline on ASR, emotion recognition, and sound captioning. Encoder replacement alone does not resolve this underuse: Whisper variants remain strongest overall, including on emotion and environmental sound captioning.

To localize the failure, we trace task-relevant information through the encoder, projector, LM layers, and LM head. Linear probes and geometric analyses show that discriminative acoustic structure remains recoverable at the final LM layer, even when MCQA accuracy trails probe accuracy by up to 83 points. Because the answer format and decoding procedure are controlled, this task-dependent gap points to content-specific readout failure rather than generic format bias. LogitLens analyses and a targeted LM head intervention support the conclusion that acoustic underuse is not explained solely by encoder-side information loss and that readout alignment can be a dominant bottleneck.

🗂️ Citation Tools & External References

(许可协议: 知识共享署名 4.0 国际)
license icon

🗂️ Citation Tools & External References

(License: Creative Commons Attribution 4.0 International)
license icon