跳转至

文章背景与核心概要

理解长达数十分钟的手术视频需要复杂且长周期的时序推理能力——例如回答关于医疗程序不同阶段之前、之后或跨阶段发生事件的问题。现有的计算机视觉方法在应对这一任务时往往捉襟见肘：一是单次输入（One-shot）的视觉语言模型（VLMs）将整个手术过程压缩到单一的上下文窗口中，从而牺牲了精确时序查询所需的细粒度视觉细节；二是传统的视频智能体需要训练模型去关注特定区域，不仅极其消耗数据，且难以泛化到域外（out-of-domain）的手术环境中。

为了克服这些局限性，本文提出了 MedClaw，这是一种模块化的智能体架构。它将高层推理与低层感知进行解耦，依靠上下文演进（context evolution）而非权重优化来实现能力提升。MedClaw 包含一个纯文本的统筹模块（orchestrator）来规划信息收集策略并发布可审计的工具调用，同时由冻结的视觉语言子智能体执行像素级任务。配合无梯度的启发式技能蒸馏机制，该系统仅需约 100 个带标签的样本就能实现高效适应，在涵盖 1,123 个经医生验证的高难度题目数据集 MedClawBench 上展现出显著的性能优势。

MedClaw: Heuristic Agent Harness for Long-Horizon Surgical Video Reasoning

arXiv ID: arXiv:2608.14015 [cs.CV]
Submitted: August 14, 2026
Authors: Yingying Fan, Penghui Du, Leyan Zhu, Runze He, Zimeng Wu, Yuxuan Zhang, Liang Chen, Jiahao Xie, Jiangtang Wang, Shuai Shao, Anchao Yang, Yutong Bai, Yan Wang
Project Page: MedClaw Project Page

Executive Summary

Understanding surgical videos that span tens of minutes requires complex, long-horizon temporal reasoning—answering questions about events that occur before, after, or across different stages of a medical procedure. Existing computer vision methodologies struggle with this task: 1. One-shot Vision-Language Models (VLMs) compress the entire surgical procedure into a single context window, sacrificing the fine-grained visual details required for precise temporal inquiries. 2. Traditional Video Agents (which train models where to look) are data-hungry and fail to generalize effectively to out-of-domain surgical environments.

To address these limitations, MedClaw introduces a modular agent harness that decouples high-level reasoning from low-level perception, relying on context evolution rather than weight optimization.

理解长达数十分钟的手术视频需要复杂且长周期的时序推理能力——例如回答关于医疗程序不同阶段之前、之后或跨阶段发生事件的问题。现有的计算机视觉方法在应对这一任务时往往捉襟见肘： 1. 单次输入视觉语言模型（One-shot VLMs） 将整个手术过程压缩到单一的上下文窗口中，从而牺牲了精确时序查询所需的细粒度视觉细节。 2. 传统视频智能体（训练模型去关注哪些区域）极为消耗数据，并且无法有效地泛化到域外的手术环境中。

为了解决这些局限性，MedClaw 引入了一种模块化的智能体架构，将高层推理与低层感知解耦，依靠上下文演进而非权重优化来实现能力提升。

Core Architecture & Methodology

  • Separation of Concerns: A text-only orchestrator plans the information-gathering strategy and issues auditable tool calls. Meanwhile, frozen vision-language sub-agents execute pixel-level tasks (e.g., viewing, cropping, inspecting specific frames, and retrieving external knowledge).
  • Heuristic Skill Distillation: A gradient-free, reward-gated feedback loop mines the agent's own low-scoring traces. It retains candidate skills exclusively when they demonstrably increase validation rewards, producing reusable retrieval techniques such as directed re-look.
  • Data Efficiency: By growing an external skill library instead of fine-tuning model weights, MedClaw successfully adapts using only ~100 labeled examples—significantly outperforming conventional supervised or reinforcement fine-tuning paradigms.
  • 职责分离： 纯文本的统筹模块（orchestrator）负责规划信息收集策略并发布可审计的工具调用；同时，冻结的视觉语言子智能体负责执行像素级任务（例如查看、裁剪、检查特定帧以及检索外部知识）。
  • 启发式技能蒸馏： 采用无梯度的、由奖励门控（reward-gated）的反馈循环来挖掘智能体自身低分轨迹中的信息。只有当候选技能能够明显提高验证奖励时才会被保留，从而产出如定向重看（directed re-look）等可重用的检索技术。
  • 数据效率： 通过扩展外部技能库而非微调模型权重，MedClaw 仅需约 100 个带标签的样本就能成功适应——显著优于常规的监督微调或强化微调范式。

MedClawBench

To rigorously evaluate long-horizon video reasoning, the authors introduce MedClawBench: * Comprises 1,123 de-leaked, doctor-grounded questions. * Evaluates performance across self-built long neurosurgery recordings and a held-out public lecture-video test split. * Results: Across both datasets and all four evaluation dimensions, MedClaw consistently outperforms one-shot VLMs and general video-agent frameworks, achieving its most significant performance gains on the challenging, out-of-domain neurosurgery videos.

为了严谨评估长程视频推理能力，作者推出了 MedClawBench： * 包含 1,123 个经过防泄漏处理、由医生标注的问题。 * 在自建的长程神经外科手术录像以及保留的公共讲座视频测试集上评估性能。 * 结果： 在两个数据集和所有四个评估维度上，MedClaw 均持续优于单次输入 VLM 以及通用视频智能体框架，并在具有挑战性的域外神经外科手术视频上取得了最显著的性能提升。