跳转至

低频陷阱：视频语言模型在简单事件记账上的失败

文章背景与核心概要

现实世界的视频基准测试往往将事件计数、速率、持续时间和视觉复杂度交织在一起，这使得研究人员很难单独隔离出模型的失败模式。与此同时，现有的程序化基准测试仅对最终答案进行评分，而没有对照真实标准（ground truth）对单个事件进行审计。为了解决这一痛点，本文作者引入了“轨迹基础的参数化分析（trace-grounded parametric profiling）”，在三种受控视频任务中评估事件计数能力。

通过对 2,190 个视频的测试，该研究揭示了当前最先进的视频语言模型（VLM，如 Gemini 3.6 Flash）中存在的阶段性时间推理失败。虽然模型能够在低频下追踪持久状态的转变，但对于瞬时事件却完全无法处理。盲目提高采样率或调整提示词只会带来表面上的准确率提升，而无法真正恢复事件本身，这证明了高计数/高频率区间仍然是当前 VLM 的根本瓶颈。

文章背景与核心概要

Real-world video benchmarks often entangle event counts, rates, durations, and visual complexities, making it difficult to isolate model failure modes. Conversely, existing programmatic benchmarks only score final answers without auditing individual events against ground truth.

To address this, the authors introduce trace-grounded parametric profiling, evaluating event counting across three controlled video tasks: bouncing-ball wall contacts, visual blinks, and categorical state transitions. Across 2,190 videos, the study reveals a staged temporal failure in state-of-the-art Video Language Models (VLMs) like Gemini 3.6 Flash. While persistent state transitions are tracked at low frequencies, transient events fail entirely. Increasing sampling rates or adjusting prompts yields superficial accuracy boosts without genuine event recovery, proving that high-count/high-frequency regimes remain a fundamental bottleneck for current VLMs.

📋 摘要

现实世界的视频基准测试虽然覆盖面广，但其固定的剪辑交织了事件计数、速率、持续时间和视觉复杂度，导致失败模式难以被隔离。虽然现有的程序化基准测试提供了更好的控制力，但它们仅对最终答案评分，而不是根据可执行的真实标准对报告的事件进行审计。

为了弥合这一差距，我们引入了轨迹基础的参数化分析，用于在三种受控视频任务中进行事件计数：弹球碰壁、视觉眨眼和分类状态转换。在 2,190 个视频中，我们在保持渲染不变的同时改变了事件计数 \(N\) 和频率 \(F\)。每个视频都包含一个可执行的事件轨迹，用于能力曲面估计和时间戳级别的评估。

我们的结果揭示了一个阶段性的时间推理失败： * 持久状态（Persistent States）： 在 80% 的可靠性阈值下，Gemini 3.6 Flash 能够可靠地计数频率在 0.5 和 1.0 Hz、事件数量多达 12 个的持久状态转换。 * 瞬时事件（Transient Events）： 对于瞬时眨眼事件，模型表现出没有可靠的正计数区域。事件表示决定了模型最初是否能访问证据——随着计数和频率的增加，这一局限性会进一步恶化。 * 高区间失败（High-Regime Failure）： 在高计数、高频率的区间中，只有 0.2% 的最终计数是正确的，并且模型仅恢复了 18.1% 的真实事件。 * 采样率限制（Sampling Rate Limitations）： 提高采样率将弹球任务的准确率从 19.6% 提升到了 29.3%，但所报告的序列与真实标准的吻合度仅为 3.7%。额外的帧数夸大了最终得分，却没有产生忠实的事件恢复。

归根结底，轨迹基础的分析将视频评估从总体准确率指标转变为对时间推理失效位置的详细诊断。

📄 Abstract

Real-world video benchmarks provide broad coverage, but their fixed clips entangle event count, rate, duration, and visual complexity, making failure modes hard to isolate. While existing programmatic benchmarks offer better control, they score only the final answer rather than auditing reported events against executable ground truth.

To bridge this gap, we introduce trace-grounded parametric profiling for event counting in three controlled video tasks: bouncing-ball wall contacts, visual blinks, and categorical state transitions. Across 2,190 videos, we vary event count \(N\) and frequency \(F\) while holding rendering fixed. Each video includes an executable event trace for capability-surface estimation and timestamp-level evaluation.

Our results reveal a staged temporal failure: * Persistent States: At an 80% reliability threshold, Gemini 3.6 Flash reliably counts persistent state transitions up to 12 events at 0.5 and 1.0 Hz. * Transient Events: Demonstrates no reliable positive-count region for transient blinking events. Event representation dictates whether a model initially accesses evidence—a limitation that compounds as count and frequency increase. * High-Regime Failure: In the high-count, high-frequency regime, only 0.2% of final counts are correct and the model recovers just 18.1% of true events. * Sampling Rate Limitations: Increasing the sampling rate boosts Bounce Ball accuracy from 19.6% to 29.3%, but the reported sequence agrees with ground truth only 3.7% of the time. Extra frames inflate final scores without producing faithful event recovery.

Ultimately, trace-grounded profiling shifts video evaluation from aggregate accuracy metrics to a detailed diagnostic of where temporal reasoning fails.

👥 作者

  • Sarvesh Baskar
  • Zikui Cai
  • Shayan Shabihi
  • Anirudh Satheesh
  • Muhammad R. Islam
  • Udari Madhushani Sehwag
  • Tom Goldstein
  • Furong Huang

👥 Authors

  • Sarvesh Baskar
  • Zikui Cai
  • Shayan Shabihi
  • Anirudh Satheesh
  • Muhammad R. Islam
  • Udari Madhushani Sehwag
  • Tom Goldstein
  • Furong Huang

🔗 链接与资源