跳转至

文章背景与核心概要

在大语言模型和智能体的对齐训练中，如何同时优化多个相互冲突或并行的目标一直是一个长期的痛点。传统的方法通常采用加权求和的方式组合多个奖励，但这往往会导致优化过程出现崩溃——模型会盲目追求最容易提升的目标，而牺牲掉其他重要的维度（例如，客服智能体变得表面热情但毫无实际帮助）。其根本原因在于，加法奖励机制缺乏对“平衡性”的考量。

为了解决这一问题，本文引入了 MINT（MIN-selection preference disTillation，最小选择偏好蒸馏）方法。MINT 对偏好蒸馏进行了极其简便的修改：它不使用奖励的加权和来对采样候选进行排序，而是根据它们的最弱目标进行排序。通过利用标准的直接偏好蒸馏（DPO）将最均衡的候选优于最失衡的候选进行蒸馏，MINT 有效解决了合作情感支持和对抗性谈判等任务中的目标崩溃问题，实现了多目标的全面提升与均衡。

MINT: Min-Selection Preference Distillation for Balanced Multi-Objective Alignment

arXiv ID: 2608.14828 [cs.AI]
Submitted: August 14, 2026
Authors: Tony Tu, Sayan Chakraborty, Ruomeng Xu, Tony Qin, Austin Tian

📌 Summary

Aligning language agents to multiple objectives simultaneously often fails when using traditional additive reward combinations. Optimization typically collapses by favoring the cheapest objective while sacrificing the rest (e.g., a support agent sounding warm while providing no actual help) due to a lack of balance in additive rewards.

This paper introduces MINT (MIN-selection preference disTillation), a simple modification to preference distillation. Instead of ranking sampled candidates using a weighted sum of rewards, MINT ranks them based on their weakest objective. By distilling the best-balanced candidate over the most lopsided one using standard Direct Preference Optimization (DPO), MINT effectively addresses objective collapse across tasks like cooperative emotional support and adversarial negotiation.

同时将语言智能体对齐到多个目标，在使用传统的加法奖励组合时往往会失败。由于加法奖励缺乏平衡性，优化过程通常会崩溃：模型会偏向于最容易优化的目标，而牺牲其余目标（例如，客服智能体听起来很热情，但实际上没有提供任何帮助）。

本文引入了 MINT（MIN-selection preference disTillation，即最小选择偏好蒸馏），这是对偏好蒸馏的一种简单修改。MINT 不使用奖励的加权和来对采样的候选进行排序，而是根据它们的最弱目标对它们进行排序。通过使用标准的直接偏好蒸馏（DPO），将最均衡的候选优于最失衡的候选进行蒸馏，MINT 有效地解决了合作情感支持和对抗性谈判等任务中的目标崩溃问题。

📝 Abstract

Aligning a language agent to several objectives at once is a persistent failure mode of preference-based training: when objectives are combined additively, optimization collapses onto whichever is cheapest to improve and sacrifices the rest, so a support agent learns to sound warm while giving no real help. The root issue is that an additive reward has no notion of balance.

We introduce Mint (MIN-selection preference disTillation), a one-line change to preference distillation: rather than ranking sampled candidates by a weighted sum of rewards, we rank them by their weakest objective, distilling the best-balanced candidate over the most lopsided one with an unchanged DPO objective. This is the \(p \to -\infty\) limit of a generalized-mean family spanning additive to worst-case selection.

Across cooperative emotional support and adversarial negotiation, min-selection lifts both objectives while sharply cutting their imbalance; on emotional support it raises the weaker axis from 0.37 to 0.64 (\(p < 10^{-40}\)), surpassing human experts and persisting across full multi-turn rollouts. A turn-by-turn analysis yields our central finding: min-selection corrects imbalance in proportion to how imbalanced the reference policy is, and its benefit endures over an interaction precisely as long as that imbalance does.

将语言智能体同时对齐到多个目标是基于偏好的训练中持续存在的一种失效模式：当目标以相加方式组合时，优化会崩溃到最容易改进的目标上并牺牲其余目标，因此支持智能体会学到听起来很温暖但毫无实际帮助。根本问题在于相加奖励没有平衡的概念。

我们引入了 Mint（MIN-selection preference disTillation，最小选择偏好蒸馏），这是对偏好蒸馏的一行代码修改：我们不是通过奖励的加权和对采样的候选进行排序，而是通过它们最弱的目标对它们进行排序，在保持 DPO 目标不变的情况下，将最平衡的候选蒸馏到最倾斜的候选之上。这是涵盖从相加到最坏情况选择的广义均值系列的 \(p \to -\infty\) 极限。

在合作情感支持和对抗性谈判中，最小选择提升了两个目标，同时大幅减少了它们的不平衡；在情感支持方面，它将较弱的轴从 0.37 提高到 0.64（\(p < 10^{-40}\)），超过了人类专家，并持续到整个多轮 Rollout 中。逐轮分析得出了我们的核心发现：最小选择对不平衡的纠正程度与参考策略的不平衡程度成正比，并且其益处在交互过程中持续的时间精确地与该不平衡持续的时间相同。