跳转至

文章背景与核心概要

本文研究了扩散模型中连接文本、音频和视频流的跨模态注意力边构成的“注意力三角形”。作者证明了这些模型中的语义泄露并非单纯的随机噪声，而是沿着这些路径进行的结构化、偏见驱动的交互结果。具体而言，音频与视频之间的连接是双向的，模型学习到的先验知识可以覆盖用户提示词，将语义重定向至视觉上符合常规但实际上并不正确的输出结果。

研究人员引入了一个诊断框架来隔离这些交互，并提出了推理时的干预方法，以在不损害生成质量的前提下改善跨模态对齐。这项研究对于提升多模态生成模型（如音视频同步生成模型）的鲁棒性和提示词遵循能力具有重要意义。

The Attention Triangle in Audio-Video Models

arXiv ID: 2609.03586
Date: September 3, 2026
Authors: Sagi Polaczek, Noa Kraicer, Gal Metzer, Zhuo Ning, Ali Mahdavi-Amiri, Daniel Cohen-Or, Raja Giryes

arXiv ID: 2609.03586
Date: September 3, 2026
Authors: Sagi Polaczek, Noa Kraicer, Gal Metzer, Zhuo Ning, Ali Mahdavi-Amiri, Daniel Cohen-Or, Raja Giryes

Summary

This paper investigates the "attention triangle"—the network of cross-modal attention edges connecting text, audio, and video streams in diffusion models. The authors demonstrate that semantic leakage in these models is not merely random noise, but a result of structured, bias-driven interactions along these pathways. Specifically, the audio-video connection is bidirectional, where learned priors can override user prompts, rerouting semantics toward visually canonical but incorrect outcomes. The researchers introduce a diagnostic framework to isolate these interactions and propose inference-time interventions to improve cross-modal alignment without compromising generation quality.

Summary

This paper investigates the "attention triangle"—the network of cross-modal attention edges connecting text, audio, and video streams in diffusion models. The authors demonstrate that semantic leakage in these models is not merely random noise, but a result of structured, bias-driven interactions along these pathways. Specifically, the audio-video connection is bidirectional, where learned priors can override user prompts, rerouting semantics toward visually canonical but incorrect outcomes. The researchers introduce a diagnostic framework to isolate these interactions and propose inference-time interventions to improve cross-modal alignment without compromising generation quality.

Key Research Findings

Key Research Findings

The Attention Triangle Mechanism

Audio-video diffusion models rely on cross-modal attention to synchronize different data streams. The study identifies that the three edges of the "attention triangle" (Text-Audio, Text-Video, and Audio-Video) act as the primary conduits for information flow.

The Attention Triangle Mechanism

Audio-video diffusion models rely on cross-modal attention to synchronize different data streams. The study identifies that the three edges of the "attention triangle" (Text-Audio, Text-Video, and Audio-Video) act as the primary conduits for information flow.

Bidirectional Semantic Leakage

The analysis reveals that the audio-video edge is inherently bidirectional: * Audio-to-Video: Audio signals influence the generation of visual content. * Video-to-Audio: Visual signals influence the generation of audio content. * Bias-Driven Routing: When a prompt conflicts with the model's internal learned priors, the cross-modal interactions often override the prompt, forcing the output toward "visually canonical" results that may be semantically inaccurate.

Bidirectional Semantic Leakage

The analysis reveals that the audio-video edge is inherently bidirectional: * Audio-to-Video: Audio signals influence the generation of visual content. * Video-to-Audio: Visual signals influence the generation of audio content. * Bias-Driven Routing: When a prompt conflicts with the model's internal learned priors, the cross-modal interactions often override the prompt, forcing the output toward "visually canonical" results that may be semantically inaccurate.

Diagnostic and Remediation Tools

Building on these insights, the authors developed: 1. Attention-Derived Signals: A method to expose how semantics are distributed and grounded across modalities. 2. Controlled Leakage Analysis: A diagnostic tool used to isolate individual interactions and understand their contribution to semantic artifacts. 3. Inference-Time Interventions: Techniques that leverage these signals to guide the model toward more consistent cross-modal alignment during the generation process.

Diagnostic and Remediation Tools

Building on these insights, the authors developed: 1. Attention-Derived Signals: A method to expose how semantics are distributed and grounded across modalities. 2. Controlled Leakage Analysis: A diagnostic tool used to isolate individual interactions and understand their contribution to semantic artifacts. 3. Inference-Time Interventions: Techniques that leverage these signals to guide the model toward more consistent cross-modal alignment during the generation process.

Access & Resources

Access & Resources

Citation

If you use this research, please cite it via the following platforms: * NASA ADS * Google Scholar * Semantic Scholar

Citation

If you use this research, please cite it via the following platforms: * NASA ADS * Google Scholar * Semantic Scholar