跳转至

谁来教授哪个 Token？面向科学推理的验证器门控多专家策略内蒸馏

Who Teaches Which Token? Verifier-Gated Multi-Expert On-Policy Distillation for Scientific Reasoning

  • arXiv ID: 2609.15404 (Submitted on 14 Sep 2026, last revised 15 Sep 2026)
  • Authors: Xun Xu, Zaixi Zhang
  • Primary Subject: Artificial Intelligence (cs.AI)
  • DOI: 10.48550/arXiv.2609.15404

文章背景与核心概要

在利用大语言模型 (Large Language Model, LLM) 攻坚科学推理难题时，如何融合多个领域专家的智慧并注入到单一模型中是极具前景的方向。然而，传统的多教师策略内蒸馏 (On-Policy Distillation, OPD) 方法往往采取粗放的“序列级别”策略，误以为专家在整条推理链条的每一个位置都能提供有益指导，反而引入了大量噪音与负迁移。针对这一痛点，本文提出了“验证器门控多专家策略内蒸馏” (Verifier-Gated Multi-Expert On-Policy Distillation, VG-OPD) 架构，通过验证器实时评估专家的真实贡献并定位认知分歧，精准解答了“究竟由哪位专家来传授哪一个 Token”的核心问题。实验表明，该方案在 4B 与 8B 参数规模的学生模型上均取得了顶尖表现，不仅稳居 7 项科学推理基准测试的综合榜首，更为复杂推理场景下的精细化知识蒸馏提供了全新范式。

执行摘要

Executive Summary

多教师策略内蒸馏 (Multi-Teacher On-Policy Distillation, OPD) 是近年来崭露头角的一种前沿方法，它旨在将各垂直领域的专家能力融会贯通到统一模型中：先利用强化学习 (Reinforcement Learning, RL) 训练出各具专长的专家模型，再以学生模型自身采样生成的推理轨迹 (Rollouts) 为基础，将专家能力蒸馏给学生模型。

Multi-teacher on-policy distillation (OPD) is an emerging approach for integrating specialist AI capabilities into a unified model by training experts using Reinforcement Learning (RL) and distilling them into a student model based on its own rollouts.

然而，传统方法通常在序列级别 (Sequence-level) 进行粗放的均匀监督分配——即把某个提示词 (Prompt) 整体分配给单一领域的专家教师，且整段回答中的每一个 Token 都被赋予相同的权重。这种做法错误地假设了专家老师在整段长回答的每一个字词上都同样有益、都能带来正向启发。

Traditional methodologies assign supervision uniformly at the sequence level (allocating each prompt to a single domain teacher where every token receives identical weighting). This approach erroneously assumes that a teacher is uniformly helpful across an entire response.

本文的研究表明，在漫长的推理轨迹上，真正有价值的教师指导信号其实是高度稀疏且参差不齐的。为了攻克这一挑战，作者团队提出了验证器门控多专家策略内蒸馏 (Verifier-Gated Multi-Expert On-Policy Distillation, VG-OPD)，通过精巧的验证机制，系统解答了“究竟谁该来教授哪一个 Token？”的核心问题：

This paper demonstrates that useful teacher signals are actually sparse and heterogeneous along a reasoning trajectory. To address this, the authors introduce Verifier-Gated Multi-Expert On-Policy Distillation (VG-OPD), which answers the question of "who should teach which token?" through a verification mechanism:

  • 反事实收益 (Counterfactual Gain)：基于具体的答案判定准则评估专家的真实贡献，以此决定是否授予该专家教学权限；
  • 分歧精确定位 (Disagreement Localization)：细致度量专家与学生之间的认知差异，精准圈定需要施加监督的关键位置；
  • 准则重要性设定 (Criterion Importance)：合理设定各评价维度的权重参数；
  • 加性优势项注入 (Additive Advantage)：将经过门控过滤的 KL 散度 (Kullback-Leibler Divergence) 作为 Token 级别的加性优势，无缝整合进分组相对策略优化 (Group Relative Policy Optimization, GRPO) 算法中。
  • Counterfactual Gain: Evaluates an expert's contribution based on specific answer criteria to authorize teaching rights.
  • Disagreement Localization: Measures divergence between the expert and student to pinpoint supervision.
  • Criterion Importance: Sets the appropriate weighting parameters.
  • Additive Advantage: Incorporates the gated KL divergence into GRPO (Group Relative Policy Optimization) as a token-level additive advantage.

核心发现与卓越性能

Key Findings & Performance

  • 全面领跑权威基准测试 (Benchmark Dominance)：在面向科学推理任务、利用强化学习训练出的能力专家进行评估时，VG-OPD 在 4B 和 8B 两种参数规模的学生模型上均取得了 7 个基准测试的综合最优成绩，并在两种规模下均夺得了 5 项基准测试的第一名。尤其在知识密集型的科学推理挑战中，该方法带来的性能飞跃最为显著。
  • 收益来源剖析 (Source of Gains)：深入分析表明，性能提升的核心动力在于对监督信号进行了经过验证的精准局部定位，而非简单地堆叠更多教师模型或微调蒸馏损失函数。错配监督预算被证实是对模型破坏性最强的做法，盲目全盘蒸馏往往会把强化学习的性能反向拖拽至基线以下——而验证器门控蒸馏机制成功遏制了这一弊端。
  • Benchmark Dominance: When instantiated for scientific reasoning using RL-trained capability experts, VG-OPD achieved the best overall performance across seven benchmarks for both 4B and 8B student models, ranking first on five benchmarks at both scales. The most significant performance gains occurred in knowledge-intensive scientific reasoning tasks.
  • Source of Gains: Analysis reveals that improvements stem from localizing verified supervision rather than simply adding more teachers or altering the distillation loss. Misplacing the supervision budget proved to be the most damaging modification, and indiscriminate distillation often dragged RL performance below its baseline—a drawback successfully mitigated by gated distillation.