跳转至

文章背景与核心概要

用于视频推理的大型多模态模型（LMMs）长期以来一直受到解析大量视觉帧所需的巨大计算开销的限制。为了解决这一问题，通常采用在线策略蒸馏（On-Policy Distillation, OPD）将大规模模型的知识迁移到更小、更高效的模型中。然而，传统的 OPD 仅匹配输出词元（token）分布，无法监督多帧视觉推理过程中形成的隐藏表征。

为了克服这一局限性，本文作者引入了 Latent-OPD，这是一种结合了轨迹级潜在蒸馏的新方法。通过对齐学生轨迹末端（此时视觉证据和上下文已得到充分总结）的隐藏状态，并利用渐进式教师前瞻策略，该模型有效地弥合了较小网络与较大网络之间的差距。实验表明，Latent-OPD 显著提升了帧效视频推理（frame-efficient video reasoning）的性能，特别是在长视频和复杂的证据聚合任务中表现尤为突出。

深思熟虑的对齐：用于视频推理的轨迹级潜在蒸馏 (Deep Thought Alignment: Trajectory-Level Latent Distillation for Video Reasoning)

arXiv: 2608.16316 [cs.CV]
提交时间: 2026年8月17日
作者: Ao Shen, Yongheng Zhang, Yinghui Li, Manning Wang, Di Yin, Xing Sun

arXiv: 2608.16316 [cs.CV]
Submitted: August 17, 2026
Authors: Ao Shen, Yongheng Zhang, Yinghui Li, Manning Wang, Di Yin, Xing Sun

📌 总结 (Summary)

大型多模态模型（LMMs）在视频推理任务中，往往受到解析海量视觉帧所需巨大计算开销的制约。为了解决这一问题，通常会使用在线策略蒸馏（On-Policy Distillation, OPD）将海量模型的知识蒸馏到更小、更高效的模型中。然而，传统的 OPD 仅匹配输出词元分布，未能对多帧视觉推理期间形成的隐藏表征进行监督。

为了解决这个问题，作者推出了 Latent-OPD，这是一种结合轨迹级潜在蒸馏的方法。通过对齐学生轨迹末端（此时视觉证据和上下文被完全总结）的隐藏状态，并利用渐进式教师前瞻策略（progressive teacher-lookahead strategy），该模型有效地弥合了小型网络与大型网络之间的差距。实验表明，Latent-OPD 显著提高了帧高效视频推理能力，特别是在长视频和复杂的证据聚合任务中。

Large Multimodal Models (LMMs) for video reasoning are often held back by the immense computational overhead required to parse vast amounts of visual frames. To address this, knowledge is typically distilled from massive models into smaller, more efficient ones using On-Policy Distillation (OPD). However, traditional OPD only matches output-token distributions and fails to supervise the hidden representations formed during multi-frame visual reasoning.

To solve this, the authors introduce Latent-OPD, a method that incorporates trajectory-level latent distillation. By aligning the hidden states at the end of student trajectories—where visual evidence and context are fully summarized—and utilizing a progressive teacher-lookahead strategy, the model effectively bridges the gap between smaller and larger networks. Experiments show that Latent-OPD significantly improves frame-efficient video reasoning, especially in long videos and complex evidence-aggregation tasks.

📋 元数据与详情 (Metadata & Details)

🔍 摘要 (Abstract)

长期以来，用于视频推理的大型多模态模型（LMMs）一直受到处理大量视觉信息的高昂计算成本的阻碍。这种困境促使人们将大模型的推理能力转移到更小、更高效的模型上。在线策略蒸馏（OPD）通过匹配学生生成轨迹沿线的输出词元分布，提供了一种有前景的解决方案。然而，视频推理通常依赖于跨多个帧积累的证据。在此背景下，输出级监督仅捕获通过词元预测表达的信息，而不能直接约束推理过程中形成的潜在表征。为了解决这一局限性，我们提出了 Latent-OPD，它通过轨迹级潜在蒸馏增强了 OPD。具体而言，我们的方法侧重于每个轨迹末端的位置，在此位置隐藏状态有效地总结了积累的视觉证据和推理上下文。此外，我们引入了渐进式教师前瞻策略，将学生模型的中后期层与逐渐加深的教师模型层进行对齐。在六个视频推理基准上的实验表明，Latent-OPD 持续优于仅输出的 OPD。值得注意的是，在帧数有限、长视频或需要复杂证据聚合的任务场景中，这种改进尤为明显。这些结果确立了 Latent-OPD 作为一种用于帧高效视频推理的高效方法的地位。

Large Multimodal Models (LMMs) for video reasoning have long been hindered by the high computational cost of processing vast amounts of visual information. This dilemma motivates the transfer of the reasoning capabilities of large models to smaller, more efficient ones. On-Policy Distillation (OPD) offers a promising solution by matching output-token distributions along student-generated trajectories. However, video reasoning often depends on evidence accumulated across multiple frames. In this context, output-level supervision only captures information expressed through token predictions and does not directly constrain the latent representations formed during reasoning. To address this limitation, we propose Latent-OPD, which augments OPD with trajectory-level latent distillation. Specifically, our method focuses on the position at the end of each trajectory, where hidden states effectively summarize the accumulated visual evidence and reasoning context. Furthermore, we introduce a progressive teacher-lookahead strategy, which aligns middle-to-late student layers with increasingly deeper teacher layers. Experiments on six video reasoning benchmarks show that Latent-OPD consistently outperforms output-only OPD. Notably, the improvements are particularly pronounced in scenarios with limited frames, long videos, or tasks requiring complex evidence aggregation. These results establish Latent-OPD as a highly effective approach to frame-efficient video reasoning.

license icon