跳转至

文章背景与核心概要

现代视觉语言模型（VLM）在处理需要细粒度视觉细节或外部知识的复杂查询时往往力不从心。为此，具备智能体特性的 VLM 引入了诸如图像裁剪、图像搜索和文本搜索等工具。然而，传统的训练范式通常仅根据最终答案的正确性来评估工具的使用情况，导致证据获取过程缺乏足够的监督。

为了解决这一局限性，本文作者引入了“必要工具-证据路径（NTEP）”标注方案以及“NTEP-R（NTEP 奖励）”监督机制。其基于 8B 参数构建的模型实例 NTEP-8B，在七个基于图像的基准测试中显著提升了面向搜索的准确率和工具使用效率。

Making Every Tool Call Count: Necessary Tool-Evidence Path Rewards for Agentic Vision-Language Models

Authors: Xingming Long, Yu Liu, Zhiwei Yang, Hanqi Feng, Shaojie Zhang, Barnabas Poczos, Chao Jiang, Zhenbo Luo, Lei Jiang, Pei Fu
Primary Subject: Artificial Intelligence (cs.AI)
arXiv ID: arXiv:2609.03493
Submitted: September 3, 2026

Executive Summary

现代视觉语言模型（VLM）在处理需要细粒度视觉细节或外部知识的复杂查询时，常常面临挑战。为了解决这一问题，智能体 VLM（Agentic VLMs）利用了诸如图像裁剪（image cropping）、图像搜索（image search）和文本搜索（text search）等工具。然而，传统的训练范式完全基于最终答案的正确性来评估工具的使用，这使得证据获取过程得不到充分的监督。

Modern vision-language models (VLMs) frequently struggle with complex queries requiring fine-grained visual details or external knowledge. To address this, agentic VLMs leverage tools such as image cropping, image search, and text search. However, traditional training paradigms evaluate tool use purely on final answer correctness, leaving evidence acquisition poorly supervised.

为了解决这一局限性，作者引入了必要工具-证据路径（NTEP, Necessary Tool-Evidence Path）标注方案以及 NTEP-R（NTEP Reward）监督机制。他们的 8B 参数模型实例 NTEP-8B 在七个基于图像的基准测试中，显著提升了面向搜索的准确率和工具使用效率。

To resolve this limitation, the authors introduce the Necessary Tool-Evidence Path (NTEP) annotation scheme and the NTEP-R (NTEP Reward) supervision mechanism. Their 8B-parameter model instance, NTEP-8B, significantly improves search-oriented accuracy and tool-use efficiency across seven image-grounded benchmarks.

Abstract

现代视觉语言模型（VLM）可以直接回答许多基于图像的问题，但在处理需要精细视觉细节或外部知识的复杂查询时，它们往往表现不佳。为了获取这些缺失的证据，智能体 VLM 会调用图像裁剪、图像 search（搜索）和文本搜索等工具。

Modern vision-language models (VLMs) can directly answer many image-grounded questions, yet they often struggle with complex queries requiring fine-grained visual details or external knowledge. To acquire this missing evidence, agentic VLMs invoke tools such as image cropping, image search, and text search.

然而，现有的训练范式主要根据最终答案的正确性来评估工具的使用，导致对证据获取和利用的监督不够充分。这带来了两个关键缺陷： 1. 模型经常发出冗余的或偏离目标的工具调用，未能收集到必要的证据。 2. 即使调用了适当的工具，模型也常常无法从产生的观察结果中提取必要的信息。

However, existing training paradigms primarily evaluate tool-use based on final answer correctness, leaving evidence acquisition and utilization insufficiently supervised. This leads to two critical shortcomings: 1. Models frequently issue redundant or off-target tool calls that fail to gather necessary evidence. 2. Even when appropriate tools are called, models often fail to extract the necessary information from the resulting observations.

为了应对这些局限性，本文引入了： * NTEP（必要工具-证据路径）： 一种新颖的标注方案，明确指定了每个查询所需的关键外部证据及相应的工具调用。 * NTEP-R（NTEP 奖励）： 一种监督机制，通过将调用前的意图和调用后的观察结果与必要的证据目标相对齐，确保每次工具调用都能严格推动推理过程朝着最终解决方案前进。 * 非重复目标正则化器（Non-Repeated-Goal Regularizer）： 一种惩罚机制，旨在防止重复访问已满足的 NTEP 目标的冗余工具调用。

To address these limitations, the paper introduces: * NTEP (Necessary Tool-Evidence Path): A novel annotation scheme that explicitly specifies the essential external evidence and corresponding tool calls for each query. * NTEP-R (NTEP Reward): A supervision mechanism ensuring that each tool invocation strictly advances the reasoning process toward the final solution by aligning pre-call intent and post-call observations with necessary evidence goals. * Non-Repeated-Goal Regularizer: A penalty mechanism designed to prevent redundant tool calls that revisit already satisfied NTEP goals.

在七个基于图像的基准测试上进行的广泛评估表明，NTEP-8B 在统一的三工具框架内，显著提升了面向搜索的准确率和工具使用效率。

Extensive evaluations on seven image-grounded benchmarks demonstrate that NTEP-8B significantly improves both search-oriented accuracy and tool-use efficiency within a unified three-tool framework.