跳转至

利用强化学习激发多模态推理代理的自我验证能力

文章背景与核心概要

在处理复杂的多模态查询时，推理代理经常因为需要解释混合的图文输入以及在稀疏的结果级监督下面对充满噪声的检索证据，而难以实现可靠的工具集成（如网络搜索）。为了解决这一痛点，本文作者引入了通过强化学习实现的自我验证（Self-Verification via Reinforcement Learning, SVRL）——这是一个纯强化学习微调框架，旨在训练代理直接在其自身的推理轨迹中验证和过滤检索到的证据。

SVRL 引入了“搜索感知惩罚”（用于抑制不必要的工具调用）以及“查询多样性奖励”（用于鼓励生成结构良好的搜索查询）。通过在仅 5,000 个视觉问答样本上使用 SVRL 对 Qwen-2.5-VL-7B 模型进行微调，该方法以极低的成本成功缩小了紧凑型代理与更大规模闭源商业模型之间的性能差距。

概览与总结 (Overview & Summary)

  • 作者： Vishwas Sathish, Viresh Ranjan, Xinliang Zhu, Arnab Dhua, Douglas Gray
  • 主要学科： 人工智能 (cs.AI)
  • 次要学科： 计算与语言 (cs.CL)、计算机视觉与模式识别 (cs.CV)
  • 会议： 将发表于 ECCV 2026（共11页正文，7幅图表）
  • arXiv ID: 2609.08025 [cs.AI]

总结

Multimodal reasoning agents frequently struggle with reliable tool integration (such as web search) due to the complexity of interpreting mixed text-image inputs and noisy evidence under sparse outcome-level supervision. To address this, the authors introduce Self-Verification via Reinforcement Learning (SVRL)—an RL-only finetuning framework that trains agents to verify and filter retrieved evidence directly within their own reasoning traces. SVRL incorporates a search-aware penalty (to discourage unnecessary tool calls) and a query-diversity reward (to encourage well-formed search queries). Fine-tuning the Qwen-2.5-VL-7B model using SVRL on only 5,000 visual question-answering examples successfully narrows the performance gap between compact agents and much larger proprietary models at a fraction of the cost.

多模态推理代理在可靠地集成工具（如网络搜索）时常常面临困难，这主要是因为在稀疏的结果级监督下，解释混合的图文输入以及处理充满噪声的检索证据具有相当的复杂性。为了解决这一问题，作者推出了通过强化学习实现的自我验证（SVRL）——这是一个纯强化学习微调框架，它训练代理直接在自身的推理轨迹中验证和过滤检索到的证据。SVRL 结合了搜索感知惩罚（用于遏制不必要的工具调用）和查询多样性奖励（用于鼓励生成格式良好且多样化的搜索查询）。仅在 5,000 个视觉问答示例上使用 SVRL 对 Qwen-2.5-VL-7B 模型进行微调，就能够在各大基准测试中实现多跳 VQA 泛化能力和工具效率的持续提升。

摘要 (Abstract)

Reasoning agents increasingly rely on external tools such as web search to answer complex queries. Reinforcement learning (RL) finetuning algorithms such as GRPO have improved long-form reasoning in text-only language models, particularly for coding and mathematics. Reliable tool use in multimodal agents, however, remains challenging because models must interpret text and images while integrating noisy retrieved evidence, often under sparse outcome-level supervision without explicit verification signals. We present Self-Verification via Reinforcement Learning (SVRL), an RL-only finetuning framework that trains multimodal agents to verify and filter retrieved evidence within their own reasoning traces, reducing reliance on external verifiers at inference time. SVRL also introduces a search-aware penalty that discourages unnecessary tool calls and a query-diversity reward that encourages diverse, well-formed search queries, providing fine-grained feedback on when and what to search. Finetuning Qwen-2.5-VL-7B with SVRL on only 5{,}000 visual question answering examples yields consistent gains in multi-hop VQA generalization and tool efficiency across benchmarks. Overall, SVRL narrows the gap between compact agents and much larger proprietary models while requiring substantially lower training and inference cost.

推理代理越来越依赖诸如网络搜索之类的外部工具来回答复杂的查询。诸如 GRPO 等强化学习（RL）微调算法已经改善了纯文本语言模型（特别是在编程和数学领域）的长文本推理能力。然而，在多模态代理中实现可靠的工具使用依然充满挑战，因为模型在缺乏显式验证信号、且通常处于稀疏结果级监督的情况下，必须同时解释文本与图像并集成充满噪声的检索证据。我们提出了通过强化学习实现的自我验证（SVRL），这是一个纯强化学习微调框架，它训练多模态代理在其自身的推理轨迹中验证和过滤检索到的证据，从而减少在推理阶段对外置验证器的依赖。SVRL 还引入了一项搜索感知惩罚（用于抑制不必要的工具调用）以及一项查询多样性奖励（用于鼓励多样化且格式良好的搜索查询），从而就“何时搜索”以及“搜索什么”提供细粒度的反馈。仅在 5,000 个视觉问答示例上使用 SVRL 微调 Qwen-2.5-VL-7B，便可在各个基准测试中稳步提升多跳 VQA 的泛化能力和工具使用效率。总体而言，SVRL 在大幅降低训练和推理成本的同时，缩小了紧凑型代理与更大规模闭源模型之间的差距。

元数据与资源 (Metadata & Resources)