跳转至

文章背景与核心概要

自我进化为预训练语言模型提供了一条可扩展的途径，使其能够在极少外部监督的情况下增强推理能力。然而，传统方法要么依赖人工精心策划的教师生成训练数据集，要么依赖启发式的难度指标，这并不能保证求解器模型获得真正的性能提升。

为了解决这些局限性，作者推出了 INFUSER，这是一个围绕两个协同演进组件构建的迭代式联合训练框架：1. 生成器（Generator）：从一组非结构化的、自动收集的文档池中草拟问题和参考标准答案；2. 求解器（Solver）：通过对生成的问题进行训练来提高其推理能力。实验表明，基于 Qwen3-8B-Base 评估，INFUSER 在奥林匹克数学和 SuperGPQA 基准测试中取得了超过 20% 的相对性能提升，展现出极强的灵活性和泛化能力。

INFUSER: Influence-Guided Self-Evolution Improves Reasoning

INFUSER: Influence-Guided Self-Evolution Improves Reasoning

arXiv: 2606.09052 [cs.LG]
Authors: Siyu Chen, Miao Lu, Beining Wu, Heejune Sheen, Fengzhuo Zhang, Shuangning Li, Zhiyuan Li, Jose Blanchet, Tianhao Wang, Zhuoran Yang
Submission History: Submitted on June 8, 2026; last revised August 21, 2026 (v4).
Links: View PDF | GitHub Repository | DOI

arXiv: 2606.09052 [cs.LG]
Authors: Siyu Chen, Miao Lu, Beining Wu, Heejune Sheen, Fengzhuo Zhang, Shuangning Li, Zhiyuan Li, Jose Blanchet, Tianhao Wang, Zhuoran Yang
Submission History: Submitted on June 8, 2026; last revised August 21, 2026 (v4).
Links: View PDF | GitHub Repository | DOI

Summary

Summary

自我进化为预训练语言模型提供了一条可扩展的途径，使其能够在极少外部监督的情况下增强推理能力。然而，传统方法要么依赖精心策划的教师生成训练数据集，要么依赖无法保证求解器模型获得真正改进的启发式难度指标。

Self-evolution offers a scalable pathway for pre-trained language models to enhance their reasoning capabilities with minimal external supervision. However, traditional approaches either rely on curated, teacher-generated training datasets or depend on heuristic difficulty metrics that do not guarantee genuine improvements in the solver model.

为了解决这些局限性，作者推出了 INFUSER，这是一个迭代式的联合训练框架，围绕两个协同且共同进化的组件构建： 1. 生成器（Generator）： 从一组非结构化的、自动收集的文档池中草拟问题和参考标准答案。 2. 求解器（Solver）： 通过对生成的问题进行训练来提升其推理能力。

To address these limitations, the authors introduce INFUSER, an iterative co-training framework built around two synergistic, co-evolving components: 1. The Generator: Drafts questions and reference golden answers sourced from a pool of unstructured, automatically collected documents. 2. The Solver: Improves its reasoning capabilities by training on the generated questions.

Key Innovations

  • 优化器感知的影响力评分： 生成器不再仅仅依赖难度启发式方法，而是通过影响力评分获得奖励，该评分决定了提出问题是否会真正改善目标分布上的求解器。
  • 双重归一化 GRPO（DuGRPO）： 为了处理影响力评分连续且本身带有噪声的特性，作者专门为生成器训练提出了一种群组相对策略优化（GRPO）的双重归一化变体。
  • 自适应课程： 这些组件共同将文档池动态转换为有针对性的课程，该课程偏爱有利于求解器当前状态的问题，而不仅仅是关注任意的难度。

Key Innovations

  • Optimizer-Aware Influence Scoring: Rather than simply relying on difficulty heuristics, the generator is rewarded using an influence score that determines whether a proposed question will genuinely improve the solver on the target distribution.
  • DuGRPO (Dual-Normalized GRPO): To handle the continuous and inherently noisy nature of the influence score, the authors propose a dual-normalized variant of Group Relative Policy Optimization (GRPO) specifically for generator training.
  • Adaptive Curriculum: Together, these components dynamically transform the document pool into a targeted curriculum that favors questions beneficial to the current state of the solver rather than just focusing on arbitrary difficulty.

Results

  • 在 Qwen3-8B-Base 上进行评估时，与强大的自我进化基线相比，INFUSER 在奥林匹克竞赛和 SuperGPQA 基准测试中实现了超过 20% 的相对提升。
  • 通过 INFUSER 共同进化的 8B 模型在数学和编程任务上的表现优于被冻结的 32B 思考生成器。
  • 消融实验证实了每个架构组件的必要性，框架的扩展（例如将 INFUSER 应用于指令微调的锚点或利用可规则验证的 RLVR 数据）证明了其强大的灵活性和泛化能力。

Results

  • Evaluated on Qwen3-8B-Base, INFUSER achieves over a 20% relative improvement on Olympiad and SuperGPQA benchmarks compared to strong self-evolution baselines.
  • An 8B model co-evolving via INFUSER outperforms a frozen 32B thinking generator on mathematics and coding tasks.
  • Ablation studies confirm the necessity of each architectural component, and framework extensions (such as applying INFUSER to an instruction-finetuned anchor or leveraging rule-verifiable RLVR data) demonstrate its strong flexibility and generalizability.

Additional Metadata

Additional Metadata

  • 学科领域（Subjects）： 机器学习 (cs.LG)；人工智能 (cs.AI)；计算与语言 (cs.CL)；计算机科学与博弈论 (cs.GT)；机器学习 (stat.ML)
  • 评论（Comments）： 67 页，17 张图表
  • 许可证（License）： 知识共享署名 4.0 （license icon 查看许可证）
  • Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computer Science and Game Theory (cs.GT); Machine Learning (stat.ML)
  • Comments: 67 pages, 17 figures
  • License: Creative Commons Attribution 4.0 (license icon view license)