跳转至

文章背景与核心概要

免训练的块稀疏注意力（Block-sparse attention）方法常被用于加速视频 Transformer 模型，然而单纯依赖按行聚合的注意力集中特性，并不能直接转化为高效且可执行的稀疏算子。本文指出了其中的关键几何瓶颈：共享块路由的 Query 往往具有重叠度很差的支撑集，且保留的注意力质量无法完全弥补因跳过交互而导致的 Softmax 后误差。

为了突破这些限制，作者推出了 SparsePR（响应耦合划分与探测拟合残差重建，Response-Coupled Partitioning with Probe-Fitted Residual Reconstruction）。该方法通过采样 Query 的 Key 响应来配对 K/V 组，并利用一小部分精确的 Query 行来校准来自稀疏输出的仿射修正（Affine correction），从而显着降低注意力重建误差。在四个异构的视频生成和世界模型上的评估表明，SparsePR 在实现 22.0%–26.0% 的实际执行对密度（Executed-pair density）的同时，保持了极高的生成质量，并实现了 1.48倍至2.61倍 的端到端加速。

Partition the Support, Reconstruct the Residual: Training-Free Sparse Attention for Video Generation and World Models

Authors: Pardis Taghavi, Reza Langari, Gaurav Pandey
Submitted: August 19, 2026
Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
arXiv: 2608.18484 [cs.CV] | DOI: 10.48550/arXiv.2608.18484
Project Page: SparsePR Website

📌 Summary

Training-free block-sparse attention methods are commonly used to accelerate video transformers, but row-wise attention concentration alone does not translate to an efficient, executable sparse operator. This paper identifies key geometric bottlenecks: queries sharing a block route often have poorly overlapping supports, and retained attention mass does not account for post-softmax errors caused by skipped interactions.

To overcome these limitations, the authors introduce SparsePR (Response-Coupled Partitioning with Probe-Fitted Residual Reconstruction). By pairing K/V groups using sampled-query key responses and utilizing a small set of exact query rows to calibrate an affine correction from the sparse output, SparsePR significantly reduces attention-reconstruction errors. Evaluated across four heterogeneous video generation and world models, SparsePR achieves 1.48x–2.61x end-to-end speedups while preserving high generation quality at a realized executed-pair density of 22.0–26.0%.

📝 摘要

免训练的块稀疏注意力可以加速视频 Transformer，但仅靠按行分布的注意力集中并不能单独确定一个可执行的稀疏算子。共享块路由的 Query 可能具有重叠度很差的支撑集，而仅靠保留的注意力质量无法决定因跳过交互而产生的 Softmax 后误差。我们表明，划分几何形状既影响汇聚的支撑集，也影响稀疏输出中剩余残差的可预测性。

我们引入了 SparsePR，它将响应耦合划分（Response-Coupled Partitioning）与探测拟合残差重建（Probe-Fitted Residual Reconstruction）结合起来。采样 Query 的 Key 响应构成了配对的 K/V 组，其质心诱导出用于共享路由的 Query-Response 坐标。随后，一小部分精确的 Query 行在探测残差所观测到的输出子空间内，对来自稀疏输出的调用特定仿射修正（Call-specific affine correction）进行校准。

在四个异构的视频 generation 和世界模型中，SparsePR 始终如一地减少了注意力重建误差。消融实验表明，探测拟合（Probe fitting）占了这种减少的大部分，而响应耦合划分降低了硬截断误差（Hard-drop error），并提高了在有限探测预算下的重建能力。SparsePR 在保持生成质量的同时，实现了 22.0%-26.0% 的实际执行对密度，并达成了 1.48x-2.61x 的端到端加速。

Training-free block-sparse attention can accelerate video transformers, but row-wise attention concentration does not by itself specify an executable sparse operator. Queries sharing a block route may have poorly overlapping supports, while retained attention mass alone does not determine the post-softmax error from skipped interactions. We show that partition geometry affects both pooled support and the predictability of the remaining residual from the sparse output.

We introduce SparsePR, which combines Response-Coupled Partitioning with Probe-Fitted Residual Reconstruction. Sampled-query key responses form paired K/V groups, whose centroids induce query-response coordinates for shared routing. A small set of exact query rows then calibrates a call-specific affine correction from the sparse output within the output subspace observed in the probe residuals.

Across four heterogeneous video generation and world models, SparsePR consistently reduces attention-reconstruction error. Ablations show that probe fitting accounts for most of this reduction, while response-coupled partitioning lowers hard-drop error and improves reconstruction under a finite probe budget. SparsePR preserves generation quality at 22.0-26.0% realized executed-pair density while achieving 1.48x-2.61x end-to-end speedups.

🛠️ SparsePR 的核心组件

  1. 响应耦合划分（Response-Coupled Partitioning）：
  2. 根据采样 Query 的 Key 响应将 Key 和 Value 分组。
  3. 诱导出用于共享路由的 Query-Response 坐标，以解决共享块路由之间支撑集重叠度差的问题。
  4. 探测拟合残差重建（Probe-Fitted Residual Reconstruction）：
  5. 采用一小部分精确的 Query 行（探测行，Probes）直接从稀疏输出中校准调用特定的仿射修正，目标对准探测残差中可见的输出子空间。
  6. 性能与效率：
  7. 降低硬截断误差。
  8. 在四个不同的视频生成和世界模型中实现了 1.48倍至2.61倍的端到端加速。
  9. 在 22.0%–26.0% 的实际执行对密度下高效运行，同时不妥协生成质量。
  1. Response-Coupled Partitioning:
  2. Groups keys and values based on sampled-query key responses.
  3. Induces query-response coordinates for shared routing to solve the issue of poorly overlapping supports among shared block routes.
  4. Probe-Fitted Residual Reconstruction:
  5. Employs a small set of exact query rows (probes) to calibrate a call-specific affine correction directly from the sparse output, targeting the output subspace visible in probe residuals.
  6. Performance & Efficiency:
  7. Lowers hard-drop errors.
  8. Delivers 1.48x to 2.61x end-to-end speedups across four diverse video generation and world models.
  9. Operates efficiently at 22.0%–26.0% realized executed-pair density without compromising generation quality.

🔗 链接与资源

License: license icon Creative Commons Attribution 4.0 International