跳转至

文章背景与核心概要

随着多模态生成技术的飞速发展，预训练的联合视听扩散模型虽然能够很好地控制生成“什么”内容，但在控制语音发生的时间（即“何时”发声）方面仍缺乏明确的机制。为了填补这一空白，本文引入了 TimeSteer 框架，通过探索去噪过程中的内在属性，实现了无需微调的推理时语音调度。

TimeSteer 的核心技术在于利用了去噪过程中的两个关键特性：一是具有时间敏感性的文本到音频交叉注意力机制，用于定位源语音的时间跨度；二是结构化的干净隐变量（clean latents），允许在不重新生成内容的情况下对音频和视觉内容的物理位置进行重定位。此外，作者团队还推出了 SpeechShift —— 这是首个专注于联合视听生成中区间级语音调度的评估基准，实验证明该方法在保持卓越生成质量的同时，大幅提升了时间可控性。

Summary

TimeSteer introduces a training-free framework for inference-time speech scheduling in joint audio-visual diffusion models. While current pretrained models offer control over what content is generated, they lack explicit temporal control over when utterances occur. TimeSteer solves this by leveraging two intrinsic properties of the denoising process: timing-sensitive cross-attention for locating source spans, and structured clean latents for repositioning content. Additionally, the authors introduce SpeechShift, the first benchmark for interval-level speech scheduling in joint audio-visual generation.

TimeSteer introduces a training-free framework for inference-time speech scheduling in joint audio-visual diffusion models. While current pretrained models offer control over what content is generated, they lack explicit temporal control over when utterances occur. TimeSteer solves this by leveraging two intrinsic properties of the denoising process: timing-sensitive cross-attention for locating source spans, and structured clean latents for repositioning content. Additionally, the authors introduce SpeechShift, the first benchmark for interval-level speech scheduling in joint audio-visual generation.

Paper Overview

论文概览

Metadata

  • Title: TimeSteer: Inference-Time Speech Scheduling in Joint Audio-Visual Diffusion Models
  • arXiv ID: 2609.01277
  • Primary Subject: Computer Vision and Pattern Recognition (cs.CV)
  • Secondary Subjects: Artificial Intelligence (cs.AI), Multimedia (cs.MM)
  • Submission Date: September 1, 2026
  • Authors: Chao Zhou, Yiling Chen, Qi Chu, Tao Gong, Nenghai Yu, Tianyi We

Metadata

  • Title: TimeSteer: Inference-Time Speech Scheduling in Joint Audio-Visual Diffusion Models
  • arXiv ID: 2609.01277
  • Primary Subject: Computer Vision and Pattern Recognition (cs.CV)
  • Secondary Subjects: Artificial Intelligence (cs.AI), Multimedia (cs.MM)
  • Submission Date: September 1, 2026
  • Authors: Chao Zhou, Yiling Chen, Qi Chu, Tao Gong, Nenghai Yu, Tianyi We

Abstract

尽管预训练的联合视听扩散模型对生成“什么”提供了丰富的控制，但它们无法对语音发生的“时间”进行显式控制。为了解决这个问题，我们研究了推理时语音调度（inference-time speech scheduling），这是一项新颖的任务，它将耦合的语音和视觉发音置于用户指定的前后时间间隔内，而无需对主干模型进行微调。

我们发现了去噪过程中的两个内在属性，使得这项任务成为可能： 1. 时间敏感的文本到音频交叉注意力头：揭示了沿隐式时间轴每个语音模型隐含的源跨度。 2. 预测的干净隐变量：已经组织好了耦合的语音和视觉发音，允许在不重新生成内容的情况下编辑它们的时间位置。

在此发现的基础上，我们提出了 TimeSteer，这是一个无需训练的框架，它通过源跨度定位（Source Span Localization）定位每个语音的源跨度，并通过区域感知隐变量重新映射（Region-Aware Latent Remapping）将相关的视听隐变量内容从源区间传输到指定的。我们进一步推出了 SpeechShift，这是联合视听生成中区间级语音调度的首个基准。在两个具有代表性的主干网络上的实验表明，与无需训练的基线相比，TimeSteer 显着提高了区间可控性，同时保持了有竞争力的整体生成质量。

Abstract

Although pretrained joint audio-visual diffusion models offer rich control over what to generate, they provide no explicit control over when an utterance should occur. To address this, we study inference-time speech scheduling, a novel task that places coupled speech and visual articulation within user-specified begin–end intervals without finetuning the backbone model.

We uncover two intrinsic properties of the denoising process that enable this task: 1. A timing-sensitive text-to-audio cross-attention head exposes each utterance's model-implied source span along the latent timeline. 2. The predicted clean latent already organizes coupled speech and visual articulation, allowing their temporal placement to be edited without regenerating the content.

Building on these discoveries, we propose TimeSteer, a training-free framework that localizes each utterance's source span through Source Span Localization and transfers the associated audio-visual latent content from the source interval to the specified target interval through Region-Aware Latent Remapping. We further introduce SpeechShift, the first benchmark for interval-level speech scheduling in joint audio-visual generation. Experiments across two representative backbones show that TimeSteer substantially improves interval controllability over training-free baselines while maintaining competitive overall generation quality.

Key Components of TimeSteer

TimeSteer 的核心组件

  • 源跨度定位（Source Span Localization）： 使用时间敏感的交叉注意力头自动跟踪并精确定位每个语音模型隐含的时间位置。
  • 区域感知隐变量重新映射（Region-Aware Latent Remapping）： 无缝移动并将相关的视听隐变量内容映射到用户定义的目标时间间隔，而无需重新训练主干模型。
  • SpeechShift 基准（SpeechShift Benchmark）： 建立第一个专为联合视听生成任务中的区间级语音调度设计的标准评估套件。

Key Components of TimeSteer

  • Source Span Localization: Automatically tracks and pinpoints the model-implied temporal location of each utterance using timing-sensitive cross-attention heads.
  • Region-Aware Latent Remapping: Seamlessly shifts and maps the associated audio-visual latent content to user-defined target time intervals without requiring backbone model retraining.
  • SpeechShift Benchmark: Establishes the first standard evaluation suite specifically designed for interval-level speech scheduling in joint audio-visual generation tasks.