跳转至

文章背景与核心概要

大型视觉语言模型（LVLMs）在处理多模态输入时，经常会出现幻觉现象，即生成与图像输入相矛盾或无关的回复。传统的缓解策略通常依赖于外部监督、输出校准或注意力机制调节，往往忽视了自回归生成过程中的内部表征动态变化。

本文指出了推理过程中的一种失效模式：跨模态表征在解码器层之间发生退化，并在不同的生成步骤中产生漂移，从而导致标记（token）预测的不稳定。为了解决这一问题，作者提出了动态对齐补偿（Dynamic Alignment Compensation, DAC），这是一种无需训练的推理时方法，能够检测表征分歧并通过以下方式应用轻量级残差补偿：1. 逐层语义补偿（Layer-wise Semantic Compensation），以抵消层间退化；2. 序列语义校正（Sequential Semantic Correction），以限制时间漂移。

在九个专注于幻觉和通用多模态基准测试上的广泛实验表明，DAC 有效地最小化了幻觉，同时保持了强大的基线性能。

Dynamic Alignment Compensation for Hallucination Mitigation in Large Vision-Language Models

Summary

Large Vision-Language Models (LVLMs) frequently suffer from hallucinations—generating responses that contradict or are irrelevant to multimodal inputs. Traditional mitigation strategies typically rely on external supervision, output calibration, or attention regulation, often overlooking internal representation dynamics during autoregressive generation.

This paper identifies an inference-time failure mode where cross-modal representations degrade across decoder layers and drift across generation steps, destabilizing token prediction. To address this, the authors propose Dynamic Alignment Compensation (DAC), a training-free inference-time method that detects representation divergence and applies lightweight residual compensation via: 1. Layer-wise Semantic Compensation to counteract inter-layer degradation. 2. Sequential Semantic Correction to restrict temporal drift.

Extensive experiments across nine hallucination-focused and general-purpose multimodal benchmarks demonstrate that DAC effectively minimizes hallucinations while preserving strong baseline performance.

Metadata & Publication Details / 元数据与出版详情

  • Authors: Kairong Yu, Zixin Zhu, Le Yu, Hongwei Wang
  • Submitted On: August 28, 2026
  • Primary Subject: Computer Vision and Pattern Recognition (cs.CV)
  • Secondary Subject: Artificial Intelligence (cs.AI)
  • Acceptance: Accepted by EMNLP 2026 Findings
  • arXiv Identifier: arXiv:2608.28058 [cs.CV]
  • DOI: 10.48550/arXiv.2608.28058

Abstract / 摘要

Large Vision-Language Models (LVLMs) remain prone to hallucinations, producing responses that are irrelevant or inconsistent with the multimodal input. Existing mitigation methods mainly rely on external supervision, output calibration, or attention regulation, leaving the internal representation dynamics of autoregressive generation underexplored. We identify an inference-time failure mode in which cross-modal representations degrade across decoder layers and drift across generation steps, destabilizing token prediction and increasing hallucination risk. We propose Dynamic Alignment Compensation (DAC), a training-free inference-time method that detects representation divergence and selectively applies lightweight residual compensation. DAC combines Layer-wise Semantic Compensation to mitigate inter-layer degradation with Sequential Semantic Correction to constrain temporal drift. Experiments on nine hallucination-focused and general-purpose multimodal benchmarks across multiple LVLM backbones show that DAC consistently reduces hallucinations while maintaining strong overall performance.

Access & Resources / 访问与资源