跳转至

我在视频中寻找你：面向以人为中心视频推理的身份条件化查询

文章背景与核心概要

现实世界的视频推理通常需要处理多模态、多源输入，然而传统的视频推理任务往往局限于简化的“视频-文本”设定，从而限制了身份匹配和以人为中心的深入分析。为了突破这一局限，本文作者提出了身份条件化查询（Identity-conditioned Queries, ICQ）任务，要求模型同时关联并解读输入视频与人物参考图像，以应对身份定位、行为理解以及时间推理等多重挑战。

为了支撑该任务，研究团队推出了 ISYV (I Seek You in Videos) 这一全面的解决方案，主要包含三个核心部分：涵盖六个难度级别的评估基准 ISYV-Bench、包含 75K 高质量样本的大规模训练集 ISYV-75K，以及能够高效利用信息丰富视频镜头的 ISYV-Framework 模组与训练策略。大量实验表明，现有的主流闭源和开源多模态大语言模型在处理跨领域身份匹配和长跨度追踪时仍显吃力，而 ISYV 模型则展现出优异的性能，为以人为中心的视频推理提供了统一的任务定义、可扩展的数据集与前沿的建模启示。

I Seek You in Videos: Identity-Conditioned Queries for Person-Centric Video Reasoning

I Seek You in Videos: Identity-Conditioned Queries for Person-Centric Video Reasoning

Summary

Summary

现实世界的视频推理通常需要解读多模态和多源输入，然而传统的视频推理任务通常依赖于简化的视频-文本设定，这限制了身份匹配和以人为中心的分析。为了解决这一局限性，作者引入了身份条件化查询（Identity-conditioned Queries, ICQ）任务，该任务要求模型将输入视频与人物的参考图像进行联合关联和解释，以处理身份定位、行为理解和时间推理。为了支持这一点，他们推出了 ISYV (I Seek You in Videos)，这是一个包含以下内容的综合解决方案： 1. ISYV-Bench：一个包含 1,377 个复杂真实世界视频及跨越六个难度级别的相应问答对的评估基准。 2. ISYV-75K：一个包含 75K 高质量样本的大规模训练集。 3. ISYV-Framework：一个面向 ICQ 的模型和训练策略，能够高效利用信息丰富的视频镜头。

Real-world video reasoning often requires interpreting multimodal and multi-source inputs, yet traditional video reasoning tasks typically rely on simplified video-text settings that limit identity matching and person-centric analysis. To address this limitation, the authors introduce the Identity-conditioned Queries (ICQ) task, which requires models to jointly associate and interpret an input video alongside a reference image of a person to handle identity grounding, behavior understanding, and temporal reasoning. To support this, they present ISYV (I Seek You in Videos), a comprehensive solution consisting of: 1. ISYV-Bench: An evaluation benchmark featuring 1,377 complex real-world videos and corresponding QA pairs spanning six difficulty levels. 2. ISYV-75K: A large-scale training set comprising 75K high-quality samples. 3. ISYV-Framework: An ICQ-oriented model and training strategy that efficiently exploits informative video shots.

Metadata & Publication Details

Metadata & Publication Details

  • arXiv ID: arXiv:2608.07417
  • 学科分类: 计算机视觉与模式识别 (cs.CV); 人工智能 (cs.AI)
  • 收录会议: 已被 ACM Multimedia 2026 (MM '26) 录用
  • 提交日期: 2026年8月7日
  • 主要参考链接: DOI
  • arXiv ID: arXiv:2608.07417
  • Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
  • Conference: Accepted to ACM Multimedia 2026 (MM '26)
  • Submission Date: August 7, 2026
  • Primary Reference: DOI

Authors

Authors

  • Shibo Gao
  • Chongxiao Wang
  • Chenglong Huang
  • Jie Ma
  • Haolin Shi
  • Fei Ding
  • Jing Li
  • Qiang Lyu
  • Yangyang Liu
  • Yang Liu
  • Jun Liu
  • Linlin Huang
  • Peipei Yang
  • Shibo Gao
  • Chongxiao Wang
  • Chenglong Huang
  • Jie Ma
  • Haolin Shi
  • Fei Ding
  • Jing Li
  • Qiang Lyu
  • Yangyang Liu
  • Yang Liu
  • Jun Liu
  • Linlin Huang
  • Peipei Yang

Abstract

Abstract

现实世界的视频推理通常涉及多模态、多源输入，而现有的视频推理任务通常假设采用简化的视频-文本设定，这限制了身份匹配和以人为中心的推理。为了弥补这一差距，我们引入了身份条件化查询（ICQ）任务，在该任务中，模型需要联合关联和解释输入视频与人物参考图像，并利用此条件来应对身份定位、行为理解和时间推理等挑战。

基于 ICQ，我们提出了 ISYV (I Seek You in Videos)，这是一个由三个组件构成的系统性解决方案： 1. ISYV-Bench，一个具有挑战性的评估基准，包含 1,377 个真实世界的复杂视频和 1,377 个问答对，分为六个难度级别，涵盖从身份识别到因果推理的能力； 2. ISYV-75K，通过自动标注、多阶段验证和人工审核构建的包含 75K 高质量样本的大规模训练集；以及 3. ISYV-Framework，包含一个面向 ICQ 的模型和训练策略，用于学习利用信息丰富的视频镜头而无需额外的镜头级标注。

广泛的实验表明，主流的闭源和开源多模态大模型（MLLM）在 ISYV-Bench 上均表现不佳，特别是在跨领域身份匹配和长跨度追踪方面。ISYV-Model 的表现优于强基线，并在某些方面接近闭源模型的性能。总的来说，ISYV 为以人为中心的视频推理提供了统一的任务定义、可扩展的数据集/基准以及建模启示。

Real-world video reasoning often involves multimodal, multi-source inputs, whereas existing video reasoning tasks typically assume a simplified video-text setting, limiting identity matching and person-centric reasoning. To bridge this gap, we introduce the Identity-conditioned Queries (ICQ) task, in which models are required to jointly associate and interpret an input video and a reference image of a person, and leverage this conditioning to address identity grounding, behavior understanding, and temporal reasoning, among other challenges.

Building on ICQ, we present ISYV (I Seek You in Videos), a systematic solution comprising three components: 1. ISYV-Bench, a challenging evaluation benchmark with 1,377 real-world complex videos and 1,377 question-answer pairs, organized into six difficulty levels spanning capabilities from identity recognition to causal reasoning; 2. ISYV-75K, a large-scale training set of 75K high-quality samples constructed via automated annotation, multi-stage verification, and manual review; and 3. ISYV-Framework, containing an ICQ-oriented model and training strategy for learning to exploit informative video shots without additional shot-level annotations.

Extensive experiments show that both mainstream closed-source and open-source MLLMs struggle on ISYV-Bench, especially in cross-domain identity matching and long-horizon tracking. ISYV-Model outperforms strong baselines and in some aspects approaches closed-source performance. Overall, ISYV provides a unified task definition, scalable datasets/benchmarks, and modeling insights for person-centric video reasoning.