跳转至

文章背景与核心概要

大视觉语言模型（LVLMs）通过将视觉感知与自然语言生成相结合，能够处理复杂的图像理解与推理任务。然而，与标准的纯文本大语言模型（LLMs）不同，LVLMs 容易产生一种独特的幻觉：生成虽然流畅但完全脱离所提供视觉输入的响应。继承自 LLMs 的现有置信度指标不足以检测这些视觉脱节问题，诊断实验表明，完全移除输入图像几乎不会改变基于置信度的选择，这证明了输出空间的置信度主要衡量的是文本合理性，而非真正的图像对齐程度。

为了克服这一局限性，作者推出了 LookBack，这是一种轻量级、无需训练的 LVLM 响应评分方法。LookBack 将传统的标记似然与视觉回溯分数（visual lookback score）相结合，该分数用于衡量生成的响应标记对底层图像标记的引用强度。在多个基准测试和模型中，LookBack 以微不足道的计算开销，持续提升了 Best-of-\(N\) 响应选择的效果。

LookBack: Where and How to Score LVLM Responses via Visual Reference Usage

Authors: Beomsik Cho, Jinhyeong Kim, Dongseok Lee, Jaehyung Kim
Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
arXiv: 2608.11847 | Submitted on: 12 August 2026
Resources: View PDF | Code Repository

Summary

大视觉语言模型（LVLMs）将视觉感知与语言生成相结合，能够处理跨越图像理解和复杂推理的响应。然而，与标准的纯文本大语言模型（LLMs）不同，LVLMs 不仅继承了文本级别的幻觉，还会针对图像产生幻觉，产生脱离其所见内容的流利响应。这使得 LVLM 响应评分在本质上更加困难，我们的诊断表明，从 LLMs 沿用下来的现有置信度指标对 LVLMs 来说是远远不够的。具体而言，移除输入图像几乎不会改变基于置信度的选择，这表明输出空间的置信度主要捕捉的是文本的合理性，而不是与图像的一致性。为了解决这一差距，我们提出了 LookBack，这是一种无需训练的 LVLM 响应评分方法，它将标记似然与视觉回溯分数（衡量每个响应标记对图像标记引用强度的轻量级指标）结合起来。在四个基准测试和三个模型中，LookBack 在几乎没有带来额外开销的情况下，持续改进了优于现有基线的 Best-of-\(N\) 选择。

Large Vision-Language Models (LVLMs) integrate visual perception with language generation, enabling responses that span image understanding and complex reasoning. However, LVLMs do not just inherit the text-level hallucinations; they also hallucinate against the image, producing fluent responses ungrounded in what they see. This makes LVLM response scoring inherently harder, and our diagnostics show that existing confidence-based metrics adopted from LLMs are insufficient for LVLMs. Specifically, removing the input image barely changes confidence-based selection, suggesting that output-space confidence primarily captures textual plausibility rather than agreement with the image. To address this gap, we propose LookBack, a training-free LVLM response scoring method that augments token likelihood with visual lookback score, a lightweight measure of how strongly each response token refers to image tokens. Across four benchmarks and three models, LookBack consistently improves Best-of-\(N\) selection over existing baselines with negligible additional overhead.

Abstract

大视觉语言模型（LVLMs）将视觉感知与语言生成相融合，能够实现涵盖图像理解与复杂推理的响应。然而，LVLMs 不仅继承了文本级幻觉，还会针对图像产生幻觉，生成脱离所见内容的流畅响应。这使得 LVLM 响应评分本质上更加困难，我们的诊断表明，从 LLMs 继承的现有基于置信度的指标对 LVLMs 而言并不充分。具体来说，移除输入图像几乎不会改变基于置信度的选择，这表明输出空间的置信度主要捕捉的是文本合理性，而非与图像的一致性。为了解决这一空白，我们提出了 LookBack，这是一种无需训练的 LVLM 响应评分方法，它通过视觉回溯分数来增强标记似然度，该分数是一个轻量级度量，用于衡量每个响应标记对图像标记的引用强度。在四个基准测试和三个模型中，LookBack 在具有微不足道的额外开销下，持续改进了 Best-of-\(N\) 选择并优于现有基线。

Large Vision-Language Models (LVLMs) integrate visual perception with language generation, enabling responses that span image understanding and complex reasoning. However, LVLMs do not just inherit the text-level hallucinations; they also hallucinate against the image, producing fluent responses ungrounded in what they see. This makes LVLM response scoring inherently harder, and our diagnostics show that existing confidence-based metrics adopted from LLMs are insufficient for LVLMs. Specifically, removing the input image barely changes confidence-based selection, suggesting that output-space confidence primarily captures textual plausibility rather than agreement with the image. To address this gap, we propose LookBack, a training-free LVLM response scoring method that augments token likelihood with visual lookback score, a lightweight measure of how strongly each response token refers to image tokens. Across four benchmarks and three models, LookBack consistently improves Best-of-\(N\) selection over existing baselines with negligible additional overhead.

Additional Information