跳转至

文章背景与核心概要

现代人工智能对齐技术通常在预训练完成之后进行，将期望的价值观和助手身份视为预先存在的行为先验之上的薄层覆盖。为了解决这种方法在应对不对齐问题时的脆弱性，作者引入了合成角色预训练（Synthetic Persona Pretraining, SPP）：一种从预训练阶段的第零个Token（token zero）开始就植入期望的助手角色的范式。

该研究通过在30亿（3B）参数规模的模型上使用5000亿（500B）Token进行预训练，证明了SPP显著改善了模型对宪章（constitution）的遵循能力、增强了越狱鲁棒性，并降低了在分布外（OOD）道德困境中的不对齐率，同时完全保留了模型的通用能力。研究还强调了早期干预的重要性，发现仅在预训练末期引入SPP会导致较弱的宪章依从性。这一优势严重依赖于角色绑定，并随着预训练计算预算的增加而表现出良好的扩展效应。

Synthetic Persona Pretraining: Alignment from Token Zero

arXiv ID: arXiv:2608.13482 [cs.LG]
Submitted: August 13, 2026
Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
Authors: Julian Minder, Viktor Moskvoretskii, Raghav Singhal, Difan Jiao, Andy Arditi, Shaobo Cui, Yiderigun Borjigin, Kartik Bali, Stefan Krsteski, Harsh Raj, Huu Nguyen, Jannik Brinkmann, Ashton Anderson, Roland Aydin, Robert West

Modern AI alignment typically occurs after pretraining, treating desired values and assistant identities as a thin overlay on top of pre-established behavioral priors. To address the vulnerability of this approach to misalignment, the authors introduce Synthetic Persona Pretraining (SPP): a paradigm that installs the desired assistant persona from token zero of the pretraining phase.

Abstract Summary

现代AI对齐通常在预训练之后进行，将期望的价值观和助手身份视为预先存在的行为先验之上的薄层覆盖。为了解决这种方法对不对齐问题的脆弱性，作者引入了合成角色预训练（Synthetic Persona Pretraining, SPP）：一种从预训练阶段的第零个Token开始就植入期望的助手角色的范式。

Abstract Summary

Modern AI alignment typically occurs after pretraining, treating desired values and assistant identities as a thin overlay on top of pre-established behavioral priors. To address the vulnerability of this approach to misalignment, the authors introduce Synthetic Persona Pretraining (SPP): a paradigm that installs the desired assistant persona from token zero of the pretraining phase.

Key Methodology

  1. Reflection Annotation: Pretraining documents are annotated with value-aligned, first-person reflections derived from a normative value constitution.
  2. Multi-Persona Pretraining: Models are pretrained using standard cross-entropy loss on standard documents alongside their reflections, establishing the desired persona among a diverse set of alternatives.
  3. Persona Binding: Post-training is conducted on user-assistant dialogue data to explicitly bind the desired persona to the core assistant identity.

Key Methodology

  1. Reflection Annotation: Pretraining documents are annotated with value-aligned, first-person reflections derived from a normative value constitution.
  2. Multi-Persona Pretraining: Models are pretrained using standard cross-entropy loss on standard documents alongside their reflections, establishing the desired persona among a diverse set of alternatives.
  3. Persona Binding: Post-training is conducted on user-assistant dialogue data to explicitly bind the desired persona to the core assistant identity.

Key Findings

  • Improved Robustness: Pretraining models up to 3B parameters on 500B tokens demonstrated that SPP significantly improves constitution-following, enhances jailbreak robustness, and decreases misalignment rates in out-of-distribution moral dilemmas, all while fully preserving general model capabilities.
  • The Importance of Early Intervention: Introducing SPP only at the tail-end of pretraining yields significantly weaker constitution adherence, fails to shift core value priorities, and results in less aligned choices during dilemmas.
  • Scaling Effect: The advantages of SPP rely heavily on persona binding and scale positively with larger pretraining budgets.

Key Findings

  • Improved Robustness: Pretraining models up to 3B parameters on 500B tokens demonstrated that SPP significantly improves constitution-following, enhances jailbreak robustness, and decreases misalignment rates in out-of-distribution moral dilemmas, all while fully preserving general model capabilities.
  • The Importance of Early Intervention: Introducing SPP only at the tail-end of pretraining yields significantly weaker constitution adherence, fails to shift core value priorities, and results in less aligned choices during dilemmas.
  • Scaling Effect: The advantages of SPP rely heavily on persona binding and scale positively with larger pretraining budgets.