跳转至

文章背景与核心概要

将大语言模型（LLM）与人类偏好进行对齐对于实际部署至关重要，但这往往会引发“对齐税”（alignment tax），导致模型预先存在的通用能力出现灾难性遗忘。现有的解决方案通常将其视为优化或架构问题，却忽视了偏好数据本身的内在品质。

为了解决这一问题，作者提出了 BALIGN，这是一种平衡的数据选择策略。通过对偏好优化梯度进行理论和实证分析，BALIGN 识别出了驱动参数漂移的三个关键数据中心特征：参考模型的对数概率裕度、胜出（chosen）与拒绝（rejected）响应之间的 Token 长度差，以及与通用能力语料库的 TF-IDF 相似度。

通过将这些特征组合成统一的风险评分，BALIGN 在保持对齐效果的同时，过滤掉了威胁基础模型参数的高风险偏好样本，且计算开销极小。这为在对齐过程中保护模型基础能力提供了一条高效且数据驱动的新途径。

Preference Data Selection for Mitigating the Alignment Tax in Large Language Models

Summary

Aligning large language models (LLMs) with human preferences often causes the alignment tax, where models experience catastrophic forgetting of their pre-existing general capabilities. Existing solutions generally treat this as an optimization or architectural problem, overlooking the intrinsic qualities of the preference data itself.

To address this, the authors propose BALIGN, a balanced data selection strategy. By analyzing the preference optimization gradient theoretically and empirically, BALIGN identifies three key data-centric features that drive parameter drift: 1. The reference model's log-probability margin. 2. The token length difference between chosen and rejected responses. 3. The TF-IDF similarity to general capability corpora.

By combining these features into a unified risk score, BALIGN filters out high-risk preference samples that threaten foundational model parameters while preserving alignment efficacy with minimal computational overhead.

Aligning large language models (LLMs) with human preferences often causes the alignment tax, where models experience catastrophic forgetting of their pre-existing general capabilities. Existing solutions generally treat this as an optimization or architectural problem, overlooking the intrinsic qualities of the preference data itself.

To address this, the authors propose BALIGN, a balanced data selection strategy. By analyzing the preference optimization gradient theoretically and empirically, BALIGN identifies three key data-centric features that drive parameter drift: 1. The reference model's log-probability margin. 2. The token length difference between chosen and rejected responses. 3. The TF-IDF similarity to general capability corpora.

By combining these features into a unified risk score, BALIGN filters out high-risk preference samples that threaten foundational model parameters while preserving alignment efficacy with minimal computational overhead.

Paper Metadata

  • arXiv ID: arXiv:2608.24192 [cs.AI]
  • Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
  • Authors: Minsu Kim, Jianxun Lian, Xing Xie, Steven Euijong Whang
  • Submission Date: August 25, 2026
  • DOI: 10.48550/arXiv.2608.24192

Paper Metadata

  • arXiv ID: arXiv:2608.24192 [cs.AI]
  • Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
  • Authors: Minsu Kim, Jianxun Lian, Xing Xie, Steven Euijong Whang
  • Submission Date: August 25, 2026
  • DOI: 10.48550/arXiv.2608.24192

Abstract

Aligning large language models to human preferences is crucial for real-world deployment but frequently incurs an alignment tax, leading to the catastrophic forgetting of pre-trained general capabilities. While previous works primarily frame this problem as an optimization or architectural challenge, the inherent characteristics of preference data that drive this degradation remain largely underexplored.

In this paper, we propose BALIGN, a balanced data selection strategy that explicitly mitigates catastrophic forgetting while optimizing alignment efficacy. Through theoretical and empirical analyses of the preference optimization gradient, we identify three key data-centric features that dictate parameter drift: * The reference model's log-probability margin, * The token length difference between chosen and rejected responses, and * The TF-IDF similarity to general capability corpora.

By aggregating these orthogonal features into a unified composite risk score, BALIGN systematically filters out high-risk preference samples that disrupt intrinsic model parameters or provide minimal alignment utility. Extensive experiments on standard human preference datasets demonstrate that BALIGN strongly preserves foundational capabilities without compromising alignment gains, consistently achieving the optimal Pareto frontier with minimal computational overhead.

Aligning large language models to human preferences is crucial for real-world deployment but frequently incurs an alignment tax, leading to the catastrophic forgetting of pre-trained general capabilities. While previous works primarily frame this problem as an optimization or architectural challenge, the inherent characteristics of preference data that drive this degradation remain largely underexplored.

In this paper, we propose BALIGN, a balanced data selection strategy that explicitly mitigates catastrophic forgetting while optimizing alignment efficacy. Through theoretical and empirical analyses of the preference optimization gradient, we identify three key data-centric features that dictate parameter drift: * The reference model's log-probability margin, * The token length difference between chosen and rejected responses, and * The TF-IDF similarity to general capability corpora.

By aggregating these orthogonal features into a unified composite risk score, BALIGN systematically filters out high-risk preference samples that disrupt intrinsic model parameters or provide minimal alignment utility. Extensive experiments on standard human preference datasets demonstrate that BALIGN strongly preserves foundational capabilities without compromising alignment gains, consistently achieving the optimal Pareto frontier with minimal computational overhead.