跳转至

文章背景与核心概要

可靠的内窥镜息肉报告需要将定量病灶测量、标准化的巴黎分级（Paris classification）以及具有临床意义的形态学描述整合到单个病历中。虽然通用的视觉语言模型（VLM）为图像理解和文本生成提供了统一的接口，但现有的专业化方法通常依赖于资源消耗巨大的任务特定模型或全模型权重微调。这些传统方法往往会损害VLM原有的预训练能力。

为了克服这一挑战，本文作者引入了一种轻量级的上下文融合框架（context-fusion framework），在不修改基础模型权重的前提下，使冻结的通用VLM实现专业化。该方法利用了两个核心组件：1. 显式转导上下文（Explicit Transduction Context）：一个自监督的息肉编码器，可检索相关的图像-报告对作为查询特定的证据；2. 隐式指令上下文（Implicit Instruction Context）：学到的连续专业令牌，用于提供跨病例共享的与具体案例无关的指令。

在包含 2,056 个经专家注释的公共内窥镜图像数据集上的评估表明，该框架显著增强了直接推理能力，在数值、分类和文本生成指标上均达到了顶级性能，同时引入的可训练参数仅占冻结VLM总大小的 0.006%。

From Generalist to Specialist: A Context-Fusion Framework for Endoscopic Polyp Reporting with a Frozen VLM

License: CC BY 4.0

  • arXiv ID: 2608.15580 [cs.AI]
  • Submitted: August 16, 2026
  • Primary Subject: Artificial Intelligence (cs.AI)
  • Secondary Subject: Computer Vision and Pattern Recognition (cs.CV)
  • Authors: Ruijie Yang, Yan Zhu, Peiyao Fu, Siyuan Li, Te Luo, Zhihua Wang, Quanlin Li, Pinghong Zhou, Xian Yang, Shuo Wang

📌 Summary

可靠的内窥镜息肉报告需要将定量的病灶大小测量、标准化的巴黎分级以及具有临床意义的形态学描述整合到单一医疗记录中。虽然通用视觉语言模型（VLM）为图像理解和文本生成提供了统一的接口，但现有的专业化方法通常依赖于资源消耗大的任务特定模型或全模型权重适应。这些传统方法往往会损害VLM的预训练能力。

为了克服这一问题，作者引入了一种轻量级的上下文融合框架，在不修改冻结的通用VLM底层权重的前提下实现其专业化。该方法利用两个核心组件： 1. 显式转导上下文： 一个自监督的息肉编码器，检索相关的图像-报告对作为查询特定的证据。 2. 隐式指令上下文： 学习到的连续专业令牌，提供跨病例共享的与具体案例无关的指令。

在 2,056 个专家注释的公开内窥镜图像上进行评估，该框架大幅增强了直接推理能力，在数值、分类和文本生成指标上均取得了顶级性能，并且引入的可训练参数仅占冻结VLM总参数量的 0.006%。

Reliable endoscopic polyp reporting requires integrating quantitative lesion sizing, standardized Paris classification, and clinically meaningful morphological descriptions into a single medical record. While general-purpose vision-language models (VLMs) offer unified interfaces for image understanding and text generation, existing approaches for specialization typically rely on resource-heavy task-specific models or full model-weight adaptations. These conventional methods often compromise the VLM's pretrained capabilities.

To overcome this, the authors introduce a lightweight context-fusion framework that specializes a frozen general-purpose VLM without modifying its underlying weights. The method leverages two core components: 1. Explicit Transduction Context: A self-supervised polyp encoder that retrieves relevant image-report pairs as query-specific evidence. 2. Implicit Instruction Context: Learned continuous specialist tokens that supply shared case-agnostic instructions.

Evaluated on 2,056 expert-annotated public endoscopic images, the framework substantially enhances direct inference, achieves top-tier performance across numeric, categorical, and text-generation metrics, and introduces trainable parameters equal to only 0.006% of the frozen VLM's total size.

🧭 Abstract

可靠的内窥镜息肉报告需要将定量的病灶测量、标准化的巴黎分级以及具有临床意义的形态学描述整合在单个记录中。通用视觉语言模型（VLM）为图像理解和报告生成提供了统一的接口。然而，现有的专业化策略通常依赖于任务特定模型或模型权重适应，这使得如何在引入可靠的专业知识的同时保持这一统一接口以及VLM的预训练能力，依然是一个未解之难题。

我们引入了一个上下文融合框架，通过隐式指令上下文和显式转导上下文，在不修改预训练权重的前提下使冻结的通用VLM专业化。具体而言，自监督息肉编码器检索相关的图像-报告对作为显式的、查询特定的证据，而学习到的连续专业令牌则提供跨病例共享的隐式指令上下文。我们在 2,056 个专家注释的公共内窥镜图像上进行了实验。我们将该框架与通用VLM、任务特定预测器和权重适应方法进行对比，以评估专业性能、统一报告生成和适应效率。在数值、分类和报告生成指标上，所提出的框架显著改进了冻结VLM的直接推理，并在所有评估方法中取得了最强的整体性能。它所增加的可训练参数仅相当于冻结VLM参数量的 0.006%。当检索到的 Top-1 病例携带正确的基准类别时，我们的框架纠正了权重适应基线所犯错误的 70.5%。这些发现支持了上下文融合框架作为冻结VLM专业化适应的一种轻量且有效的策略。

Reliable endoscopic polyp reporting requires integrating quantitative lesion sizing, standardized Paris classification, and clinically meaningful morphological description within a single record. General-purpose vision-language models (VLMs) offer a unified interface for image understanding and report generation. Existing specialization strategies, however, typically rely on task-specific models or model-weight adaptation, leaving unresolved how to introduce reliable specialist knowledge while preserving both this unified interface and the VLM's pretrained capabilities.

We introduce a context-fusion framework that specializes a frozen general-purpose VLM through both implicit instruction context and explicit transduction context without modifying its pretrained weights. Specifically, a self-supervised polyp encoder retrieves related image-report pairs as explicit, query-specific evidence, while learned continuous specialist tokens provide implicit instruction context shared across cases. Experiments were conducted on 2,056 expert-annotated public endoscopic images. We compared the framework with general-purpose VLMs, task-specific predictors, and weight-adaptation methods to assess specialist performance, unified reporting, and adaptation efficiency. Across numerical, categorical, and report-generation metrics, the proposed framework substantially improved direct frozen-VLM inference and achieved the strongest overall performance among the evaluated methods. It added trainable parameters equal to only 0.006% of the frozen VLM's parameter count. When the top-1 retrieved case carried the correct target category, our framework corrected 70.5% of the errors made by a weight-adaptation baseline. These findings support the context-fusion framework as a lightweight and effective strategy for specialist adaptation of a frozen VLM.

🔗 Quick Links & Full-Text Access

快速链接与全文访问