跳转至

文章背景与核心概要

本文探讨了大语言模型（LLM）安全对齐的核心脆弱性。当前的许多大模型依赖于事后对齐技术（如 RLHF 和 DPO），但这些技术极易遭到越狱、微调攻击和探针探测的破解。作者通过分析权重更新与模型曲率之间的几何关系，揭示了事后安全本质上只是在原有能力之上叠加了一个“拒绝门”，而并未真正擦除潜在的危险能力。

在此基础上，研究进一步通过对 OLMo-2-1B 等模型在预训练全周期的多检查点追踪，发现安全性构成的结构性基质在预训练的特定阶段会发生急剧转变。研究证明，如果在预训练阶段持续引入安全协同训练，模型将获得强大的结构鲁棒性，能够有效抵御各种规模下的对抗攻击，并显著优于传统的事后对齐方案。

The Geometry of Refusal: Why Post-Hoc Safety Is Fragile and Pretraining-Time Safety Persists

拒绝的几何学：为什么事后安全机制如此脆弱，而预训练阶段的安全机制能够持久

Authors: Srikanth Malla, Chiho Choi, Joon Hee Choi
Submitted: September 7, 2026
Subjects: Machine Learning (cs.LG), Artificial Intelligence (cs.AI)
arXiv: 2609.06934 | DOI: 10.48550/arXiv.2609.06934

Executive Summary

执行摘要

Post-hoc alignment techniques like RLHF and DPO are standard for making large language models safe, yet they remain vulnerable to jailbreaks, fine-tuning attacks, and probing. This paper demonstrates that post-hoc safety updates act as a "refusal gate" laid over intact capabilities rather than erasing them. By analyzing the geometry of these updates against model curvature, the authors reveal why post-hoc safety is inherently fragile—and show that integrating safety continuously throughout pretraining offers robust, persistent protection against adversarial attacks across various model scales.

诸如 RLHF 和 DPO 之类的事后对齐技术是确保大语言模型安全性的标准方法，但它们仍然容易受到越狱、微调攻击以及探测技术的威胁。本文证明，事后安全更新实际上是在完好的模型能力之上叠加了一个“拒绝门”，而并未真正将其擦除。通过分析这些更新相对于模型曲率的几何形态，作者揭示了事后安全机制本质上为何如此脆弱，并表明在整个预训练过程中持续整合安全机制，能够为不同规模的模型提供抵御对抗攻击的强大且持久的保护。

Key Findings & Geometric Analysis

核心发现与几何分析

1. The Suppression Regime of Post-Hoc Safety

1. 事后安全的抑制机制

When measuring the safety weight update \(\Delta = W_{\text{safe}} - W_{\text{base}}\) against the curvature of the model's capabilities (the empirical Fisher of a capability loss): * Orthogonality: Post-hoc safety updates consistently land nearly orthogonal to capability directions. * Sharp Focus: Their small in-subspace components concentrate heavily on a few high-curvature directions. * Result: The update functions as a thin, sharp "refusal gate" masking underlying capabilities rather than erasing them.

当我们将安全权重更新量 \(\Delta = W_{\text{safe}} - W_{\text{base}}\) 与模型能力的曲率（能力损失的经验费雪信息矩阵）进行对比测量时发现： * 正交性： 事后安全更新的方向与能力方向始终近似正交。 * 尖锐聚焦： 它们在子空间内的微小分量高度集中于少数几个高曲率方向。 * 结果： 这种更新的作用类似于一个薄而尖锐的“拒绝门”，它只是掩盖了底层能力，而并没有将其抹去。

2. The Kernel-Immobility Lemma

2. 核不动性引理

Because post-hoc updates merely mask rather than remove capabilities: * Benign fine-tuning can effortlessly restore the original behavior. * Example: Just 100 steps of benign fine-tuning completely collapse refusal mechanisms in Qwen-2.5-7B and Llama-3-8B Instruct while preserving overall capabilities—a vulnerability that replicates across five major model families.

由于事后更新仅仅是掩盖而非移除了这些能力： * 良性微调可以轻而易举地恢复模型的原始行为。 * 示例： 仅需 100 步的良性微调，就能彻底瓦解 Qwen-2.5-7B 和 Llama-3-8B Instruct 中的拒绝机制，同时却完好地保留了其整体能力——这种漏洞在五个主流模型系列中均可复现。

3. Pretraining-Time Safety Emergence

3. 预训练阶段安全性的涌现

Tracing safety back into the pretraining phase via a 267-checkpoint sweep of OLMo-2-1B: * The structural substrate that safety engages emerges via a sharp transition between 6B and 60B pretraining tokens. * Constructive Outcome: Models trained from scratch with continuous safety co-training throughout pretraining achieve 87% to 98% refusal rates. * Post-attack refusal levels remain resilient at 84% to 91% across every scale (an erosion of only 2 to 14 pp, compared to 35 to 38 pp for post-hoc methods), maintaining capability-matched or superior performance from 410M up to 6.9B parameters.

通过对 OLMo-2-1B 的 267 个检查点进行全盘扫描，将安全性追溯至预训练阶段： * 安全机制所依赖的结构性基质在预训练词数达到 60亿到600亿之间 时，通过一次急剧转变涌现出来。 * 建设性成果： 在整个预训练过程中从头开始进行持续安全协同训练的模型，其拒绝率达到了 87% 至 98%。 * 在遭受攻击后，各规模模型的拒绝率仍能保持在 84% 至 91% 的稳健水平（性能下降仅为 2 到 14 个百分点，而事后方法的降幅高达 35 到 38 个百分点），并且在从 410M 到 6.9B 参数的范围内，保持了与原版相匹配甚至更优越的性能。

Conclusion

结论

The persistence of model safety relies fundamentally on the persistence of the safety signal across pretraining, rather than the timing of its implementation. Co-training models continuously throughout pretraining creates structural robustness that superficial post-hoc installs simply cannot match.

模型安全性的持久性根本上取决于安全信号在预训练全过程中的持续存在，而不是其实施的时机。在整个预训练过程中对模型进行持续的协同训练，能够创造出表面化的事后部署根本无法企及的结构鲁棒性。

Access Paper & Resources

访问论文与资源