跳转至

连续扩散语言模型在规模化扩展上可媲美离散扩散

Continuous Diffusion Scales Competitively with Discrete Diffusion for Language

arXiv:2605.18530 [cs.CL]
Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Machine Learning (stat.ML)
Authors: Zhihan Yang, Wei Guo, Shuibai Zhang, Subham Sekhar Sahoo, Yongxin Chen, Arash Vahdat, Morteza Mardani, John Thickstun
Submitted: 18 May 2026; Last revised: 9 Sep 2026 (v2)
DOI / Paper: 10.48550/arXiv.2605.18530

文章背景与核心概要

在图像生成领域大放异彩的扩散模型，近年来被广泛探索用于自然语言建模任务。然而，长期以来学界普遍认为连续扩散在文本领域难以规模化扩展，性能往往落后于离散扩散方案。

本文重新审视了基于似然估计的连续扩散语言模型 (Diffusion Language Model, DLM) Plaid，通过将其网络架构与现代主流离散 DLM 对齐，构建了全新的 RePlaid 模型。该研究首次确立了能够直接媲美离散扩散模型的连续 DLM 扩展法则 (Scaling Law) ，将连续扩散与自回归模型之间的算力开销差距显著缩小至仅 \(20\times\) ，并在 OpenWebText 数据集上取得了 22.1 的连续 DLM 最优困惑度 (Perplexity, PPL) 上界与卓越的生成质量。研究从理论上揭示了优化噪声调度以最小化 ELBO 方差能自然带来跨时间维度的均匀信息衰减，有力地证明了连续扩散在语言建模上兼具极高竞争力与扩展前景。

论文元数据

Metadata

  • arXiv 标识符： arXiv:2605.18530 [cs.CL]
  • 学科分类： 计算与语言 (cs.CL)；人工智能 (cs.AI)；机器学习 (cs.LG)；统计机器学习 (stat.ML)
  • 作者列表： Zhihan Yang, Wei Guo, Shuibai Zhang, Subham Sekhar Sahoo, Yongxin Chen, Arash Vahdat, Morteza Mardani, John Thickstun
  • 提交历程： 首次提交于 2026年5月18日；最新修订于 2026年9月9日 (v2)
  • 链接资源： 查看 PDF | HTML 版本 | DOI 索引
  • arXiv ID: arXiv:2605.18530 [cs.CL]
  • Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Machine Learning (stat.ML)
  • Authors: Zhihan Yang, Wei Guo, Shuibai Zhang, Subham Sekhar Sahoo, Yongxin Chen, Arash Vahdat, Morteza Mardani, John Thickstun
  • Submission History: Submitted on 18 May 2026; Last revised 9 Sep 2026 (v2).
  • Links: View PDF | HTML Version | DOI

核心执行概要

Executive Summary

虽然扩散模型近期在自然语言处理社区引发了广泛关注，但在可扩展性 (Scalability) 方面，连续扩散长期以来普遍被认为明显逊色于离散方案。本研究打破了这一固有偏见，重新审视了基于似然的经典连续扩散语言模型 (DLM) Plaid ，并提出了将 Plaid 架构与现代离散 DLM 紧密对齐的全新变体 RePlaid 。

While diffusion models have gained significant traction in language modeling, continuous diffusion has traditionally lagged behind discrete approaches in terms of scalability. This paper challenges that limitation by revisiting Plaid—a likelihood-based continuous diffusion language model (DLM)—and introducing RePlaid, which aligns Plaid's architecture with modern discrete DLMs.

该工作的核心发现与主要贡献包括： * 媲美离散的可扩展性： RePlaid 建立了首个在规模化扩展上与离散 DLM 旗鼓相当的连续 DLM 扩展法则 (Scaling Law) ，其与自回归模型之间的算力消耗差距仅为 \(20\times\) 左右。 * 更为优越的模型性能： RePlaid 仅需更少的参数量即可击败 Duo ，并在过度训练 (Over-trained) 机制下全面超越 MDLM 。 * 顶尖基准表现： 在 OpenWebText 基准评测中，RePlaid 在连续 DLM 中斩获了全新的最优困惑度 (PPL) 上界 22.1 ，并展现出更出色的生成文本质量。 * 深刻的理论洞见： 作者证明了通过优化噪声调度以最小化证据下界 (ELBO) 方差，会自然形成随时间呈线性衰减的交叉熵（信息损失），从而无需针对特定案例进行复杂的时间重参数化即可均匀分散去噪难度。此外，研究发现通过似然目标优化词嵌入能够构建出结构化的几何空间，成为带来显著似然增益的核心驱动力。

Key findings and contributions of this work include: * Competitive Scaling: RePlaid establishes the first scaling law for continuous DLMs that closely rivals discrete models, showing a compute gap of only \(20\times\) compared to autoregressive models. * Superior Performance: RePlaid outperforms Duo using fewer parameters and surpasses MDLM in the over-trained regime. * State-of-the-Art Results: Evaluated on OpenWebText, RePlaid achieves a new state-of-the-art Perplexity (PPL) bound of 22.1 among continuous DLMs along with superior generation quality. * Theoretical Insights: The authors demonstrate that optimizing the noise schedule to minimize ELBO variance naturally results in linear cross-entropy (information loss) over time, evenly distributing denoising difficulty without case-specific time reparameterizations. Furthermore, likelihood-based embedding optimization creates structured geometries that drive significant likelihood gains.

摘要

Abstract

尽管扩散机制近期吸引了语言建模领域的极大关注，但连续扩散在规模扩展性上此前似乎一直落后于离散方法。为了挑战这一观点，我们重新探讨了基于似然驱动的连续扩散语言模型 (DLM) Plaid，并通过将 Plaid 的网络架构与现代离散 DLM 进行对齐，构建了 RePlaid。在这一统一评测体系下，我们确立了首个足以匹敌离散 DLM 的连续 DLM 扩展法则：RePlaid 与自回归模型相比仅存在 \(20\times\) 的算力差距，在使用更少参数的情况下战胜了 Duo，并在过度训练机制下超越了 MDLM。我们在最新的连续 DLM 基准上对 RePlaid 进行了评估：在 OpenWebText 上，RePlaid 实现了 22.1 的全新最佳 PPL 上界，并展现出更为优越的生成质量。这些结果表明，当采用似然目标进行训练时，连续扩散是离散 DLM 极具竞争力且具备强扩展性的替代方案。此外，我们提供了理论视角来阐释似然训练的优势。我们证明，优化噪声调度以最小化 ELBO 的方差会自然带来跨时间的线性交叉熵（信息损失），这无需任何特定用例的时间重参数化即可平稳分布去噪难度。同时，我们发现通过似然优化词嵌入能形成富有规律的几何结构，并带来最为显著的似然收益。

While diffusion has drawn considerable recent attention from the language modeling community, continuous diffusion has appeared less scalable than discrete approaches. To challenge this belief we revisit Plaid, a likelihood-based continuous diffusion language model (DLM), and construct RePlaid by aligning the architecture of Plaid with modern discrete DLMs. In this unified setting, we establish the first scaling law for continuous DLMs that rivals discrete DLMs: RePlaid exhibits a compute gap of only \(20\times\) compared to autoregressive models, outperforms Duo while using fewer parameters, and outperforms MDLM in the over-trained regime. We benchmark RePlaid against recent continuous DLMs: on OpenWebText, RePlaid achieves a new state-of-the-art PPL bound of \(22.1\) among continuous DLMs and superior generation quality. These results suggest that continuous diffusion, when trained via likelihood, is a highly competitive and scalable alternative to discrete DLMs. Moreover, we offer theoretical insights to understand the advantage of likelihood-based training. We show that optimizing the noise schedule to minimize the ELBO's variance naturally yields linear cross-entropy (information loss) over time. This evenly distributes denoising difficulty without any case-specific time reparameterization. In addition, we find that optimizing embeddings via likelihood creates structured geometries and drives the most significant likelihood gain.

(与本文相关的版权声明图标： license icon 查看许可协议)

(License icon associated with this article: license icon View License)