跳转至

探究自动驾驶轻量级视觉语言模型中的视觉概念探测

文章背景与核心概要

视觉语言模型（VLMs）在自动驾驶领域的应用日益广泛，旨在通过先进的推理和泛化能力来处理长尾场景。然而，这些模型在驾驶相关的基础视觉任务上经常失效，其根本原因目前仍大体不明。

在这项工作中，作者检查了五个最先进（SOTA）轻量级VLM的中间激活，以识别信息瓶颈。通过结合反事实图像集与线性探测（linear probing）技术，该研究揭示了：对象或主体存在等概念被显式且线性地编码；而空间概念（如方向）则仅由视觉编码器的空间布局隐式捕获，或完全缺乏线性编码。此外，研究指出了两种关键的失效模式（感知失效与认知失效），并发现随着目标物体距离的增加，视觉概念的线性可分性会迅速退化。

📌 摘要 (Summary)

Vision-Language Models (VLMs) are increasingly applied in automated driving to manage long-tail scenarios through advanced reasoning and generalization. However, these models frequently fail on fundamental visual tasks relevant to driving, and the root causes remain largely unclear.

In this work, the authors examine the intermediate activations of five state-of-the-art (SOTA) lightweight VLMs to identify information bottlenecks. By analyzing counterfactual image sets paired with linear probing techniques, the research reveals: * Explicit Encoding: Concepts such as the presence of an object or agent in a scene are explicitly and linearly encoded. * Implicit or Missing Encoding: Spatial concepts like object/agent orientation are only implicitly captured by the vision encoder's spatial layout or lack linear encoding entirely. * Two Key Failure Modes: 1. Perceptual Failure: Required visual information is absent from the model's linear activations. 2. Cognitive Failure: Information is present, but the model fails to correctly align it with language semantics. * Distance Degradation: The linear separability of visual concepts degrades rapidly as the distance of the target object increases.

📄 文章元数据 (Article Metadata)

Field Details
arXiv ID arXiv:2603.06054 [cs.CV]
Subjects Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
Journal Reference Transactions on Machine Learning Research, 2026
Submission History • v1: March 6, 2026
• v2 (Last Revised): August 6, 2026
License Creative Commons Attribution 4.0 International

👥 作者 (Authors)

  • Nikos Theodoridis
  • Reenu Mohandas
  • Ganesh Sistu
  • Anthony Scanlan
  • Ciarán Eising
  • Tim Brophy

🔗 全文与资源 (Full-Text & Resources)

license icon