跳转至

文章背景与核心概要

长期以来，控制语言模型的情感基调通常依赖于离散的标签（如“高兴”、“愤怒”、“悲伤”），这难以捕捉诸如“略带沮丧但保持平静”等细腻的情感状态。为了解决这一痛点，本文提出了 VA-DPO（Valence-Arousal Direct Preference Optimization）框架，利用效价-唤醒度（Valence-Arousal, VA）连续平面 \((v^*, a^*)\) 来实现精确的情感生成控制。

该方法对直接偏好优化（DPO）进行了微小而巧妙的调整：通过冻结一个 VA 回归器来计算生成文本与目标情感之间的欧氏距离，进而筛选偏好数据。在 Llama-3.1-8B-Instruct 模型上的实验表明，该方法将目标 VA 距离的均值比系统提示词（system-prompting）降低了 33%，比少样本提示词（few-shot prompting）降低了 25%，同时将效价和唤醒度的相关性分别提升至 \(r_v = 0.93\) 和 \(r_a = 0.75\)。更重要的是，这种精细的情感控制并未以牺牲模型核心能力（如 MMLU、HellaSwag 和 TruthfulQA）为代价，展现了极高的实用价值。

VA-DPO: Valence-Arousal Direct Preference Optimization for Controllable Emotion Generation in Language Models

  • Authors: Hyunwoo Kim
  • Subject: Computation and Language (cs.CL)
  • arXiv ID: arXiv:2608.20374
  • Submission Date: June 23, 2026

📌 Summary

Controlling the emotional tone of language models is traditionally handled through discrete labels (e.g., happy, angry, sad), which fail to capture nuanced emotional states like "mildly downcast but calm."

To address this, VA-DPO introduces a continuous emotion generation framework using the Valence-Arousal (VA) plane \((v^*, a^*)\). By making a small adjustment to Direct Preference Optimization (DPO)—where preference data is curated using a frozen VA regressor to measure Euclidean distance to a target emotion—the method successfully aligns language models with precise emotional specifications without degrading core capabilities like MMLU, HellaSwag, or TruthfulQA.

Controlling the emotional tone of language models is traditionally handled through discrete labels (e.g., happy, angry, sad), which fail to capture nuanced emotional states like "mildly downcast but calm."

To address this, VA-DPO introduces a continuous emotion generation framework using the Valence-Arousal (VA) plane \((v^*, a^*)\). By making a small adjustment to Direct Preference Optimization (DPO)—where preference data is curated using a frozen VA regressor to measure Euclidean distance to a target emotion—the method successfully aligns language models with precise emotional specifications without degrading core capabilities like MMLU, HellaSwag, or TruthfulQA.

📖 Abstract

How precisely can we tell a language model how to feel? Most work on emotional generation answers with a discrete label - happy, angry, sad - which cannot express a target like "mildly downcast but calm." We instead specify the desired affect as a continuous point \((v^*, a^*)\) in the Valence-Arousal plane and train the model to hit it.

Our method, VA-DPO, is a small modification to Direct Preference Optimization: 1. A frozen VA regressor scores each sampled generation by its Euclidean distance to the target. 2. We keep only candidate pairs whose distance gap clears a margin \(\tau\). 3. We optimize a LoRA adapter with the ordinary DPO loss against a frozen reference.

The DPO objective itself is unchanged; what is new is how the preference data is built. On Llama-3.1-8B-Instruct this cuts mean VA distance to the target by 33% over system-prompting and 25% over few-shot prompting, lifting valence/arousal correlation to \(r_v = 0.93\) and \(r_a = 0.75\). The gains carry over to Qwen3-8B and Llama-3.2-3B, and they do not come at the usual price: MMLU is unchanged (\(\Delta = +0.0\)) and HellaSwag and TruthfulQA are preserved.

We release the code, configs, and the preference-construction pipeline.

How precisely can we tell a language model how to feel? Most work on emotional generation answers with a discrete label - happy, angry, sad - which cannot express a target like "mildly downcast but calm." We instead specify the desired affect as a continuous point \((v^*, a^*)\) in the Valence-Arousal plane and train the model to hit it.

Our method, VA-DPO, is a small modification to Direct Preference Optimization: 1. A frozen VA regressor scores each sampled generation by its Euclidean distance to the target. 2. We keep only candidate pairs whose distance gap clears a margin \(\tau\). 3. We optimize a LoRA adapter with the ordinary DPO loss against a frozen reference.

The DPO objective itself is unchanged; what is new is how the preference data is built. On Llama-3.1-8B-Instruct this cuts mean VA distance to the target by 33% over system-prompting and 25% over few-shot prompting, lifting valence/arousal correlation to \(r_v = 0.93\) and \(r_a = 0.75\). The gains carry over to Qwen3-8B and Llama-3.2-3B, and they do not come at the usual price: MMLU is unchanged (\(\Delta = +0.0\)) and HellaSwag and TruthfulQA are preserved.

We release the code, configs, and the preference-construction pipeline.