跳转至

文章背景与核心概要

在大语言模型（LLM）的后训练阶段，在线蒸馏（On-Policy Distillation, OPD）一直是核心技术之一。然而，本文指出了一种关键的失效模式：退化一致性（Degenerate Agreement）。在这种状态下，学生模型学会利用重复循环来与教师模型达到较高的Token级一致性，即便其底层响应在全局上存在严重缺陷。

为此，作者提出将研究重点从单纯的Token一致性转移到解决师生不匹配（teacher-student mismatch）上来，并将其分为两种主要类型：学生多余Token（Student-excess tokens）和学生缺失Token（Student-deficit tokens）。为了解决这些问题，作者引入了 TIDE（Token级独立缺失-多余修正，Token-level Independent Deficit-Excess correction） 算法。TIDE采用有界黑林格塑形（bounded Hellinger shaping）来抑制严重的多余项，并利用解析教师top-\(K\)注入来恢复缺失项的概率质量，而无需强制采样这些Token。在数学推理基准上的实验结果表明，TIDE显著优于标准OPD和现有基线，尤其是在师生不匹配严重的场景下。

Mismatch Matters: On-Policy Distillation Beyond Token Agreement

arXiv: 2608.09836
Date: August 10, 2026
Authors: Zichao Yu, Chengzhi Yu, Shengze Xu, Yujin Han, Bingqing Jiang, Xu Wang, Difan Zou

arXiv: 2608.09836
Date: August 10, 2026
Authors: Zichao Yu, Chengzhi Yu, Shengze Xu, Yujin Han, Bingqing Jiang, Xu Wang, Difan Zou

Summary

On-policy distillation (OPD) is a cornerstone of modern Large Language Model (LLM) post-training. However, this paper identifies a critical failure mode: degenerate agreement. In this state, student models learn to exploit repetitive loops to achieve high token-level agreement with the teacher, even when the underlying responses are globally flawed.

The authors propose shifting the focus from simple token agreement to addressing teacher-student mismatch, categorized into two primary types: * Student-excess tokens: Tokens generated by the student that the teacher assigns near-zero probability. These cause unstable, unbounded log-ratio corrections. * Student-deficit tokens: Tokens preferred by the teacher but rarely sampled by the student, which prevent the effective transfer of reasoning patterns.

To resolve these issues, the authors introduce TIDE (Token-level Independent Deficit-Excess correction). TIDE employs bounded Hellinger shaping to suppress severe excesses and uses an analytic teacher top-\(K\) injection to restore probability mass for deficits without requiring those tokens to be sampled. Experimental results on mathematical reasoning benchmarks show that TIDE significantly outperforms standard OPD and existing baselines, particularly in scenarios with high teacher-student mismatch.

Summary

On-policy distillation (OPD) is a cornerstone of modern Large Language Model (LLM) post-training. However, this paper identifies a critical failure mode: degenerate agreement. In this state, student models learn to exploit repetitive loops to achieve high token-level agreement with the teacher, even when the underlying responses are globally flawed.

The authors propose shifting the focus from simple token agreement to addressing teacher-student mismatch, categorized into two primary types: * Student-excess tokens: Tokens generated by the student that the teacher assigns near-zero probability. These cause unstable, unbounded log-ratio corrections. * Student-deficit tokens: Tokens preferred by the teacher but rarely sampled by the student, which prevent the effective transfer of reasoning patterns.

To resolve these issues, the authors introduce TIDE (Token-level Independent Deficit-Excess correction). TIDE employs bounded Hellinger shaping to suppress severe excesses and uses an analytic teacher top-\(K\) injection to restore probability mass for deficits without requiring those tokens to be sampled. Experimental results on mathematical reasoning benchmarks show that TIDE significantly outperforms standard OPD and existing baselines, particularly in scenarios with high teacher-student mismatch.

Key Contributions

  • Identification of Degenerate Agreement: A novel analysis of why current OPD methods fail by prioritizing token-level mimicry over semantic reasoning.
  • Mismatch Categorization: A formal distinction between "excess" and "deficit" tokens, providing a clearer framework for understanding distillation failures.
  • TIDE Algorithm: A robust correction mechanism that:
    • Applies bounded Hellinger shaping to stabilize updates against excess tokens.
    • Utilizes analytic teacher top-\(K\) injection to guide the student toward better reasoning paths.
  • Performance Gains: Demonstrated improvements in mathematical reasoning, including a jump in Avg@8 from 6.9% to 20.3% and a 3.6x reduction in average response length under high-mismatch conditions.

Key Contributions

  • Identification of Degenerate Agreement: A novel analysis of why current OPD methods fail by prioritizing token-level mimicry over semantic reasoning.
  • Mismatch Categorization: A formal distinction between "excess" and "deficit" tokens, providing a clearer framework for understanding distillation failures.
  • TIDE Algorithm: A robust correction mechanism that:
    • Applies bounded Hellinger shaping to stabilize updates against excess tokens.
    • Utilizes analytic teacher top-\(K\) injection to guide the student toward better reasoning paths.
  • Performance Gains: Demonstrated improvements in mathematical reasoning, including a jump in Avg@8 from 6.9% to 20.3% and a 3.6x reduction in average response length under high-mismatch conditions.

Access & Resources

Access & Resources