跳转至

大规模遗忘：数十亿参数语言模型中的状态精确且保留轨迹的删除

文章背景与核心概要

在大模型时代，如何从已训练的模型中彻底抹去特定数据（即“机器遗忘”）是一个备受关注且极具挑战性的课题。传统的遗忘方法往往难以在保证模型性能的同时，实现数学或状态层面的精确删除。本文探讨了在从回放数据集中移除选定样本后，通过预先埋点的训练延续是否能够准确复制删除的反事实结果。

作者引入了一种“保留轨迹的反事实方法”（trace-preserving counterfactual），该方法在保持记录的执行控制不变的同时，将所请求删除的标识符贡献置零。研究在多种架构（包括 Pythia 160M、Pythia 2.8B 和 Llama 3.2 1B）上进行了测试，证明了在固定单 GPU 环境下数十亿参数的状态精确性。然而，该研究也指出，这一过程并不等于“低成本删除”，因为分散的删除请求可能会迫使模型进行接近全量的重放。

license icon

Summary

This paper investigates whether prospectively instrumented training continuations can accurately replicate deletion counterfactuals after removing selected examples from a replay dataset. The author introduces a trace-preserving counterfactual approach that keeps recorded execution controls constant while assigning zero contribution to requested identifiers.

Testing across various architectures—including Pythia 160M, Pythia 2.8B, and Llama 3.2 1B—demonstrates billion-parameter state exactness under pinned single-GPU environments. However, the study notes that this process does not equate to "cheap deletion," as dispersed requests can force near-full replays. Standardized TOFU/OpenUnlearning measurements are provided strictly as descriptive diagnostics.

论文元数据

Paper Metadata

字段 详情
arXiv 标识符 arXiv:2508.12220 [cs.LG]
作者 Abdullah X
主要学科 机器学习 (cs.LG)
次要学科 人工智能 (cs.AI)、密码学与安全 (cs.CR)
提交时间 2025年8月17日
最后修订 2026年8月12日 (v2)
DOI 10.48550/arXiv.2508.12220
代码与研究记录 GitHub 仓库 (v0.3.1)
Field Details
arXiv Identifier arXiv:2508.12220 [cs.LG]
Authors Abdullah X
Primary Subject Machine Learning (cs.LG)
Secondary Subjects Artificial Intelligence (cs.AI), Cryptography and Security (cs.CR)
Submitted On August 17, 2025
Last Revised August 12, 2026 (v2)
DOI 10.48550/arXiv.2508.12220
Code & Research Records GitHub Repository (v0.3.1)

摘要

预先埋点的训练延续能否在选定的样本离开其回放数据集后，准确重现删除的反事实（counterfactual）结果？我们研究了一种保留轨迹的反事实方法，该方法固定了记录的执行控制，同时为请求的标识符分配零贡献。这种保证是前瞻性的：原始运行必须记录此执行溯源（provenance）并保留符合条件的未污染检查点。在固定的单 GPU 环境下，从一个未包含请求行的词元存储（token store）中重放数据，可以在模型和优化器状态上逐位（bit-for-bit）重构出单独执行的轨迹神谕（trace oracle）。Pythia 160M 在四种删除几何结构下均表现出精确性；Pythia 2.8B 在随机 5% 的请求下匹配了所有 2,775,208,960 个模型状态元素；而 Llama 3.2 1B 在从回放存储中省略了 4,000 个 TOFU 示例中的 400 个之后也达到了精确。这些结果确立了数十亿参数的状态精确性。它们并未确立低成本删除，因为分散的请求可能会迫使进行几乎全量的重放。我们发布标准化 TOFU/OpenUnlearning 测量结果仅作为描述性诊断，因为冻结的实验缺乏因果行为声明所需的匹配控制。

Abstract

Can a prospectively instrumented training continuation reproduce a deletion counterfactual exactly after selected examples leave its replay dataset? We study a trace-preserving counterfactual that fixes recorded execution controls while assigning requested identifiers zero contribution. The guarantee is prospective: the original run must record this execution provenance and retain an eligible uncontaminated checkpoint. Under pinned single-GPU environments, replay from a token store materialized without the requested rows reconstructs a separately executed trace oracle bit-for-bit in model and optimizer state. Pythia 160M is exact across four deletion geometries; Pythia 2.8B matches all 2,775,208,960 model-state elements for a random 5% request; and Llama 3.2 1B is exact after omitting 400 of 4,000 TOFU examples from replay storage. These results establish billion-parameter state exactness. They do not establish cheap deletion, because dispersed requests can force nearly full replay. We release standardized TOFU/OpenUnlearning measurements as descriptive diagnostics only because the frozen campaign lacks the matched controls required for a causal behavioral claim.

获取与资源

Access & Resources