跳转至

文章背景与核心概要

当前的深度学习后训练（Post-training）方法普遍认为，模型的高效学习必须依赖于密集的Token监督，对海量的Token进行优化。然而，这篇来自最新arXiv论文的研究对这一传统假设发起了挑战。作者们在策略内蒸馏（On-Policy Distillation, OPD）框架下进行研究，意外地发现：仅通过极小比例的生成Token（每个推理轨迹仅需1到2个Token，约占总Token数的0.05%），就能有效地激发和提升大模型的推理能力。

通过在Qwen3系列模型、数学与代码推理任务、Llama模型以及基于可验证奖励的强化学习（RLVR）上的广泛验证，研究表明这种稀疏监督策略在效果上完全可以媲美甚至超越全Token训练。这种现象与人类自然学习的过程高度相似——人类往往是通过对关键节点进行反思和总结，而非对每个字进行微观纠错。该研究颠覆了“后训练必须依赖大量Token”的传统认知，为开发更高效的大模型后训练算法指明了新的方向。

极度稀疏的监督信号即可激发推理能力 (Extremely Sparse Supervision Incentivizes Reasoning Ability)

arXiv ID: 2609.04565 [cs.AI]
提交日期: 2026年9月3日
作者: Zhishuai Liu, Xingzi Xu, Mehmet Saygin Seyfioglu, Pan Xu, Karim Bouyarmane

📌 总结 (Summary)

Prevailing post-training methods for large language models typically assume that effective learning must be token-intensive, optimizing over massive numbers of tokens. This paper challenges that assumption by examining the on-policy distillation (OPD) setting.

当前主流的大语言模型后训练方法通常假设：有效的学习必须是密集消耗Token的，需要在海量的Token上进行优化。本文通过考察策略内蒸馏（on-policy distillation, OPD）的设定，对这一假设发起了挑战。

Using the Qwen3 family, the authors discover a counter-intuitive phenomenon: reasoning capabilities can be effectively incentivized using an extremely small fraction of generated tokens—as few as one or two tokens per trajectory (roughly 0.05% of all tokens). Surprisingly, this sparse supervision matches or outperforms full-token training across mathematical and coding reasoning tasks, Llama models, and Reinforcement Learning with Verifiable Reward (RLVR) via PPO. This approach mirrors natural human learning more closely: reflecting on critical milestones rather than performing micro-corrections at every single word.

利用Qwen3系列模型，作者们发现了一个反直觉的现象：仅使用极小一部分生成的Token——每个轨迹只需1到2个Token（大约占所有Token的0.05%）——就能有效地激发推理能力。令人惊讶的是，在数学和代码推理任务、Llama模型以及通过PPO实现的基于可验证奖励强化学习（RLVR）中，这种稀疏监督的效果能够匹配甚至超越全Token训练。这种方法更贴近人类的自然学习过程：对关键的里程碑进行反思，而不是对每一个字进行微观纠错。

📋 文献信息 (Bibliographic Information)

📑 摘要 (Abstract)

Large language models demonstrate increasingly strong reasoning capabilities through effective post-training. Yet, prevailing post-training methods optimize over massive numbers of tokens, implicitly assuming that effective learning must be token-intensive. We revisit this assumption in the on-policy distillation (OPD) setting, which naturally admits dense teacher supervision at every generated token. Using the Qwen3 family, we discover a counter-intuitive phenomenon: reasoning can be effectively incentivized by an extremely small fraction of generated tokens--as few as one or two tokens per reasoning trajectory, corresponding to only 0.05% of all tokens. Surprisingly, this sparse supervision in most cases matches or surpasses full-token training in improving reasoning ability, despite excluding the vast majority of generated tokens from the training objective. This phenomenon is consistently observed across nine teacher--student configurations spanning different model scales on mathematical reasoning tasks, and is further validated on coding reasoning, Llama models and Proximal Policy Optimization (PPO)-based reinforcement learning with verifiable reward (RLVR). Interestingly, such extremely sparse supervision may be closer to the natural learning process: rather than correcting every step word by word, one reflects on a few critical reasoning steps, updates prior understanding, and continues the trial-and-error, avoiding micro-level corrections while remaining remarkably effective. Overall, our results challenge the assumption that effective post-training must be token-intensive and point to a new direction for understanding and designing more efficient post-training algorithms.

大语言模型通过有效的后训练展现出日益强大的推理能力。然而，现有的后训练方法在海量Token上进行优化，隐式地假设了有效的学习必须是密集消耗Token的。我们在策略内蒸馏（on-policy distillation, OPD）的设定下重新审视了这一假设，该设定原本自然地允许对每个生成的Token提供密集的教师监督。利用Qwen3系列模型，我们发现了一个反直觉的现象：推理能力可以通过极小比例的生成Token来有效激发——每个推理轨迹只需1到2个Token，仅占所有Token的0.05%。令人惊讶的是，尽管在训练目标中排除了绝大多数生成的Token，这种稀疏监督在大多数情况下在提升推理能力方面仍能匹配甚至超越全Token训练。这一现象在跨越不同模型规模的九种教师-学生配置下的数学推理任务中得到了持续观察，并在代码推理、Llama模型以及基于近端策略优化（PPO）的可验证奖励强化学习（RLVR）中得到了进一步验证。有趣的是，这种极度稀疏的监督可能更贴近人类的自然学习过程：与其逐字纠正每一步，不如反思几个关键的推理步骤，更新先前的理解，并继续试错，从而避免了微观层面的纠错，同时保持了显著的效果。总的来说，我们的结果挑战了“高效后训练必须消耗大量Token”的假设，并为理解和设计更高效的后训练算法指明了新的方向。