跳转至

文章背景与核心概要

本文聚焦于流匹配语言模型（Flow-matching language models）在少步采样（few-step sampling）下面临的质量急剧下降问题。传统方法中，基于前序预测的自条件机制（previous-prediction self-conditioning）在训练和推理阶段存在本质上的“训练-推理不匹配”（train–inference mismatch），这会导致潜空间与自条件输入之间产生不断累积的冗余，从而严重损害生成质量。

为了在无需重新训练模型的前提下解决这一痛点，作者提出了全新的非绑定自条件机制（Untied Self-Conditioning）采样器。该方法包含两大核心创新：一是利用冻结的投影权重识别并抑制自条件输入与潜空间高度冗余的方向（潜空间冗余阻尼）；二是根据求解器的积分结构推导并近似所需的步平均预测，其缩放比例由离线轨迹统计数据决定。实验表明，该方法在保持每步仅需一次评估的高效性的同时，大幅提升了少步采样下的生成质量，在多个基准测试中展现出显著的性能优势。

Improving Few-Step Language Flows with Untied Self-Conditioning

Authors: Bocheng Li, Linli Xu
Published: 23 August 2026
Primary Subject: Computation and Language (cs.CL)
Other Subjects: Artificial Intelligence (cs.AI), Machine Learning (cs.LG)
Identifiers: arXiv:2608.22244 [cs.CL] | DOI: 10.48550/arXiv.2608.22244

Summary

流匹配语言模型能够并行精炼所有Token位置，允许从业者通过牺牲采样步数来换取更低的延迟。然而，传统上当仅使用少数采样步数时，生成质量会急剧下降。

本文指出这一退化的根本原因在于先前预测自条件机制中固有的训练-推理不匹配（train–inference mismatch）： * 在训练期间： 自条件输入是直接从当前的噪声状态计算得出的，没有任何介入的求解器步骤。 * 在采样期间： 求解器在完全相同的预测作为显式自条件输入重新出现之前，就已经将前一个预测整合到了潜状态中。

这种耦合会产生不断增长的冗余，从而损害自条件输入和求解器的更新。为了在无需重新训练模型的情况下解决这个问题，作者引入了非绑定自条件机制（Untied Self-Conditioning），这是一种每个步骤仅需一次评估的新型采样器： * 潜空间冗余阻尼（Latent Redundancy Dampening）： 利用冻结的投影权重，识别并阻尼自条件输入与潜空间相冗余的方向。 * 步平均修正（Step-Average Correction）： 从预测历史中近似理论上必需的步平均预测，并通过离线轨迹统计进行缩放。

Flow-matching language models enable the parallel refinement of all token positions, allowing practitioners to trade sampling steps for reduced latency. However, generation quality traditionally degrades sharply when using only a few sampling steps.

This paper identifies the root cause of this degradation as a train–inference mismatch inherent to previous-prediction self-conditioning: * During training: The self-conditioning input is computed directly from the current noisy state without any intervening solver steps. * During sampling: The solver integrates the previous prediction into the latent state before that exact prediction reappears as an explicit self-conditioning input.

This coupling creates a growing redundancy that impairs both the self-conditioning input and the solver update. To resolve this without requiring model retraining, the authors introduce Untied Self-Conditioning, a novel sampler requiring only one evaluation per step: * Latent Redundancy Dampening: Identifies and dampens directions where the self-conditioning input is redundant with the latent space using frozen projection weights. * Step-Average Correction: Approximates the theoretically necessary step-average prediction from the prediction history, scaled via offline trajectory statistics.

Abstract

流匹配语言模型可以并行精炼所有Token的位置，并能够通过采样步数来权衡延迟，但在使用少量采样步数时，生成质量仍然会急剧下降。我们将这种退化的源头追溯到先前预测自条件机制中的训练-推理不匹配：在训练期间，自条件输入是从当前的噪声状态计算得出的，没有介入的求解器步骤；在采样期间，求解器在相同的预测作为显式自条件输入重新出现之前，就将先前的预测折叠到了潜空间中。这种在训练期间不存在的耦合会产生随步长增长的冗余。我们证明了这种不匹配会损害自条件输入和求解器的更新，并从模型自身的结构中为每项推导出了修正方法。从冻结的投影权重中，我们识别并阻尼了自条件输入与潜空间冗余的方向；从求解器的积分结构中，我们推导得出需要步平均预测，并从预测历史中对其进行近似，其尺度由离线轨迹统计数据设定。由此产生的采样器——非绑定自条件机制，无需重新训练，且每步使用一次评估。在 LangFlow 的 8 个采样步骤下，它将 OpenWebText 的生成困惑度从 \(531\) 降低到 \(62\)（提升 \(8.6\) 倍）；在改编后的 Arena-Hard-Auto v2 协议下，其输出在 96% 的成对比较中受到青睐。在 ELF-B 上，它将生成困惑度从 \(71\) 降低到 \(43\)。性能改进在 8 到 256 个采样步骤的范围内均保持稳定。

Flow-matching language models refine all token positions in parallel and can trade sampling steps for latency, yet generation quality still degrades sharply with few sampling steps. We trace a source of this degradation to a train--inference mismatch in previous-prediction self-conditioning: during training, the self-conditioning input is computed from the current noisy state with no intervening solver step; during sampling, the solver folds the previous prediction into the latent before that same prediction reappears as the explicit self-conditioning input. This coupling, absent during training, creates redundancy that grows with step width. We show that the mismatch degrades both the self-conditioning input and the solver update, and derive a correction for each from the model's own structure. From the frozen projection weights we identify directions along which the self-conditioning input is redundant with the latent and dampen them; from the solver's integration structure we derive that a step-average prediction is needed and approximate it from prediction history, with scale set by offline trajectory statistics. The resulting sampler, Untied Self-Conditioning, requires no retraining and uses one evaluation per step. At 8 sampling steps on LangFlow, it reduces OpenWebText generative perplexity from \(531\) to~\(62\) (\(8.6\times\)); under an adapted Arena-Hard-Auto~v2 protocol, its outputs are preferred in \(96\%\) of pairwise comparisons. On ELF-B it reduces generative perplexity from \(71\) to~\(43\). Improvements hold from 8 to 256 sampling steps.

Key Results

  • LangFlow（8个采样步）： 将 OpenWebText 的生成困惑度从 531 降至 62，实现了 \(8.6\times\) 的性能提升。
  • Arena-Hard-Auto v2 协议： 在成对比较中，由非绑定自条件机制生成的输出有 96% 的概率受到青睐。
  • ELF-B： 将生成困惑度从 71 降至 43。
  • 鲁棒性： 在 8 到 256 个采样步的扫描范围内，性能增益保持一致。
  • LangFlow (8 Sampling Steps): Reduces OpenWebText generative perplexity from 531 to 62 an \(8.6\times\) improvement.
  • Arena-Hard-Auto v2 Protocol: Outputs generated by Untied Self-Conditioning are preferred in 96% of pairwise comparisons.
  • ELF-B: Reduces generative perplexity from 71 to 43.
  • Robustness: Performance gains remain consistent across sweep ranges from 8 to 256 sampling steps.

Access Paper & Resources