跳转至

文章背景与核心概要

随着大语言模型在推理任务中的广泛应用，基于可验证奖励的强化学习（RLVR）成为提升模型能力的核心技术之一。然而，由 Renfei Zhang 和 Niloofar Mireshghallah 发表的这项最新研究揭示了 RLVR 一个长期被忽视的副作用：在完全良性、不包含任何隐私的客观事实数据上执行强化学习，会显著增加模型对先前在预训练阶段暗中记忆的个人身份信息（PII）的提取率。

研究表明，虽然指令模型经常在预训练期间默默记忆 PII，但在常规提示下极难将其唤醒。然而，通过在完全无关的良性事实数据上进行微调，模型对 PII 的提取率会大幅飙升。例如，在 DeepSeek-V3.1 模型上，逐字复述率增加了 \(2.4\) 倍（从 \(0.155\) 升至 \(0.370\)），且绝对泄露量随着模型参数规模（从 8B 到 671B）的增加而扩大。更为关键的是，模型的通用推理能力和拒绝率在此过程中保持不变，这表明强化学习选择性地“解锁”了对已记忆私密数据的访问权限，而非广泛破坏模型的整体性能。

这一发现带来了深刻的安全启示：攻击者无需任何与隐私相关的训练信号或直接的数据访问权限，仅需通过无害的良性信息对模型进行微调，便能诱导模型泄露内部记忆的隐私数据，对当前大模型的隐私合规与安全部署提出了全新的挑战。

Reinforcement Learning on Benign Facts Amplifies Leakage of Memorized Private Data

Authors: Renfei Zhang, Niloofar Mireshghallah
Submitted: August 22, 2026
Subjects: Machine Learning (cs.LG), Artificial Intelligence (cs.AI)
arXiv: 2608.21727 | DOI: 10.48550/arXiv.2608.21727

📌 Summary

本文研究了常用于增强模型推理任务的“基于可验证奖励的强化学习（RLVR）”的一个被忽视的副作用。作者证明，在完全良性、非隐私的事实数据上执行强化学习，会显著增加对模型已经秘密记忆的个人身份信息（PII）的提取。

This paper investigates an overlooked side effect of Reinforcement Learning with Verifiable Rewards (RLVR), commonly used to enhance model reasoning tasks. The authors demonstrate that performing reinforcement learning on completely benign, non-private factual data significantly increases the extraction of Personally Identifiable Information (PII) that the model has already secretly memorized.

Key Findings:

  • Latent Memorization: Instruct models frequently memorize PII during pre-training, but rarely surface it under normal prompting.
  • The Amplification Effect: Fine-tuning via RL on entirely benign factual data (devoid of PII) drastically increases PII extraction across both targeted probes (e.g., name-to-email pairs) and untargeted free-recall prompts.
  • Scale and Impact: On DeepSeek-V3.1, verbatim recall increases by \(2.4\times\) (from \(0.155\) to \(0.370\)). Across models spanning from 8B to 671B parameters, absolute leakage scales with model size.
  • Selective Accessibility: General reasoning abilities and refusal rates remain intact, indicating that RL selectively unlocks access to memorized private data rather than broadly disrupting the model.

Security Implication: An adversary can extract memorized private data without requiring any privacy-relevant training signals or direct data access—merely by fine-tuning the model on innocuous, benign information.