跳转至

文章背景与核心概要

大型视觉语言模型（LVLMs）在广泛的任务中取得了显著性能，但它们往往会从训练数据中继承社会偏见，在处理来自不同社会群体的个人肖像时产生有偏见的行为。传统的去偏技术在解码过程中通常依赖于单一的刻板视角来比较标记概率。为了解决这一局限性，研究人员提出了反事实集成解码（Counterfactual Ensemble Decoding, CED）方法。

该研究受社会科学中“多样性能促进公平”这一原则的启发，CED 在视觉表征空间中构建多群体反事实视角，并在解码过程中将它们进行集成，以促进模型的公平行为。该框架在跨职业、描述符和角色特征等场景下实现了显著的偏见降低（最高达 47.97%），同时完美保留了模型的核心能力。

通过反事实集成解码减轻大型视觉语言模型中的偏见

摘要 (Summary)

大型视觉语言模型（LVLMs）经常从其训练数据中继承社会偏见，在处理来自不同社会群体的个人肖像时导致有偏见的输出。传统的去偏技术通常在解码过程中通过比较标记（token）概率来依赖单一的刻板视角。为了解决这一局限性，研究人员提出了反事实集成解码（CED）。受社会科学中“多样性促进公平”这一原则的启发，CED 在视觉表征空间内构建多群体反事实视角，并在解码过程中进行集成，以促进公平的模型行为。该框架实现了显着的偏见减少（在职业、描述符和角色特征方面最高可达 47.97%），同时保留了模型的核心能力。

Large Vision-Language Models (LVLMs) frequently inherit social biases from their training data, leading to biased outputs when processing portraits of individuals from diverse social groups. Traditional debiasing techniques often rely on a single stereotyped viewpoint by comparing token probabilities during decoding. To address this limitation, researchers propose Counterfactual Ensemble Decoding (CED). Inspired by the social science principle that diversity fosters fairness, CED constructs multi-group counterfactual perspectives within the visual representation space and integrates them during decoding to promote equitable model behavior. The framework achieves substantial bias reduction (up to 47.97% across occupations, descriptors, and persona traits) while preserving the model's core capabilities.

元数据 (Metadata)

作者 (Authors)

  • Yisong Xiao
  • Aishan Liu
  • Yongxin Huang
  • Zonghao Ying
  • Shiji Zhao
  • Tianlin Li
  • Yong Han
  • Jian Yang
  • Xianglong Liu

摘要详述 (Abstract)

大型视觉语言模型（LVLMs）在广泛的任务中取得了卓越的性能；然而，它们经常从训练数据中继承社会偏见，在处理来自不同社会群体的肖像时导致有偏见的行为。

Large Vision-Language Models (LVLMs) have achieved remarkable performance across a wide range of tasks; however, they often inherit social biases from their training data, resulting in biased behavior when processing portraits from different social groups.

现有的去偏方法通常在解码过程中比较原始生成和有偏生成之间的标记概率，但它们根本上的局限性在于依赖单一的刻板视角，无法考量社会观点的多样性。受社会科学中“多样性促进公平”这一原则的启发，我们提出了反事实集成解码（CED），这是一个新颖的框架，它在视觉表征空间内构建多群体反事实视角，并在解码过程中将其集成，以促进公平的模型行为。

Existing debiasing approaches typically compare token probabilities between the original and biased generations during decoding, but they are fundamentally limited by their reliance on a single, stereotyped viewpoint and fail to account for the diversity of social perspectives. Inspired by the social science principle that diversity fosters fairness, we propose Counterfactual Ensemble Decoding (CED), a novel framework that constructs multi-group counterfactual perspectives within the visual representation space and integrates them during decoding to promote equitable model behavior.

CED 首先通过识别与每个社会群体相关的语义方向并在这些方向上生成反事实表征，在视觉空间中执行反事实引导，从而提供打破刻板印象叙事的多元视角。在解码过程中，CED 定位出在这些视角间表现出最大差异的解码器层，并使用不确定性感知权重对它们的标记分布进行集成，优先考虑来自不同群体的具有高置信度的标记，从而产生更平衡的概率分布，以指导更公平的生成。

CED first performs counterfactual steering in the visual space by identifying semantic directions associated with each social group and generating counterfactual representations along these directions, thereby offering diverse perspectives that disrupt stereotypical narratives. During decoding, CED locates the decoder layer exhibiting the greatest divergence among these perspectives and ensembles their token distributions using uncertainty-aware weights, prioritizing high-confidence tokens from different groups to yield a more balanced probability distribution that guides fairer generation.

在三个社会偏见评估基准上进行的广泛实验表明，与领先的基线相比，CED 实现了显著的性能提升，在涉及职业、描述符和角色特征的场景中将偏见降低了高达 47.97%。此外，CED 还能以极小的性能下降保留原始模型的核心能力。

Extensive experiments on three social bias evaluation benchmarks demonstrate that CED achieves substantial improvements over leading baselines, reducing bias by up to 47.97% across scenarios involving occupations, descriptors, and persona traits. Moreover, CED also preserves the core capabilities of the original model with minimal degradation.

访问与资源 (Access & Resources)

外部参考与引用 (External References & Citations)