服务掩码扩散大语言模型：来自真实硬件的特征分析与设计原则

文章背景与核心概要

随着大语言模型（LLM）的不断演进，掩码扩散语言模型（dLLMs）作为一种能够同时对多个Token进行去噪的新型范式，正逐渐成为传统自回归（AR）模型的有力替代方案。然而，现有的推理与服务基础设施大多基于自回归模型的假设构建，难以完全适应dLLMs独特的运行行为。为了填补这一空白，本文以运行在NVIDIA H200 GPU上的 LLaDA-8B-Instruct 模型及 D2F（Discrete Diffusion Forcing） LoRA适配器为测试对象，对dLLM的服务性能进行了首次全面的特征分析。

研究发现，dLLM请求所需的去噪步数呈现出独特的离散特征，且在生成开始前极难准确预测，这给预生成调度带来了挑战。此外，研究指出GPU计算仅占总端到端耗费时间的24%，其余76%均归因于CPU端的调度开销，而通过批处理（Batching）则能极大地摊销这一开销。基于上述发现，本文提出了面向未来扩散大语言模型推理系统的核心设计原则，强调必须在单个去噪步数的粒度上实现并行化，并重新构思调度算法与批处理超时规则。

摘要 (Summary)

掩码扩散语言模型（dLLMs）通过同时对多个Token进行去噪，为传统的自回归（AR）模型提供了一种极具潜力的替代方案。然而，当前的服务基础设施往往依赖于源自AR模型的假设，这些假设可能并不适用于dLLMs的独特行为。

本文使用搭载 D2F（Discrete Diffusion Forcing） LoRA适配器的 LLaDA-8B-Instruct 模型在NVIDIA H200 GPU上，对dLLM服务进行了首次全面的特征分析。作者识别出了优化吞吐量和延迟的关键瓶颈与设计原则，并得出结论：高效的dLLM服务需要在单个去噪步数的粒度上实现并行化。

Masked diffusion language models (dLLMs) offer a promising alternative to traditional autoregressive (AR) models by denoising multiple tokens simultaneously. However, current serving infrastructure often relies on assumptions derived from AR models that may not apply to the unique behavior of dLLMs.

This paper provides the first comprehensive characterization of dLLM serving using the LLaDA-8B-Instruct model with a D2F (Discrete Diffusion Forcing) LoRA adapter on an NVIDIA H200 GPU. The authors identify key bottlenecks and design principles to optimize throughput and latency, concluding that effective dLLM serving requires parallelism at the level of individual denoising steps.

核心发现 (Key Findings)

1. 离散的请求难度

研究人员发现，请求所需的去噪步数并不是一个连续变量，而是落入11个固定的离散级别中。值得注意的是，在生成开始之前，并没有可靠的信号来预测所需的步数级别（最佳 \(R^2 = 0.150\)），这使预生成调度变得非常复杂。

1. Discrete Request Difficulty

The researchers discovered that the number of denoising steps required for a request is not a continuous variable but falls into 11 fixed, discrete levels. Notably, there is no reliable signal to predict the required step-count level prior to the start of generation (best \(R^2 = 0.150\)), complicating pre-generation scheduling.

2. 基准测试的局限性

具有较短生成预算（320个Token以下）的标准基准测试大大低估了服务延迟的方差。这些基准测试通常在扩散过程固有的延迟离散度显现之前，就提前截断了请求。

2. Benchmarking Limitations

Standard benchmarks with short generation budgets (under 320 tokens) significantly understate the variance in serving latency. These benchmarks often truncate requests before the true latency spread—inherent to the diffusion process—can manifest.

3. GPU与CPU开销对比

对于单个请求而言，总端到端墙钟时间中只有24%消耗在GPU计算上。其余76%归因于CPU端的调度开销。 * 优化方案： 批处理（Batching）在摊销此项开销方面极为高效。 * 性能表现： 与单请求调度基准相比，采用16的批大小（Batch Size）可将吞吐量提升 16.0倍。

3. GPU vs. CPU Overhead

Only 24% of the total wall-clock time for a single request is spent on GPU computation. The remaining 76% is attributed to CPU-side dispatch overhead. * Optimization: Batching is highly effective at amortizing this overhead. * Performance: Using a batch size of 16 improves throughput by 16.0x compared to a per-request-dispatch baseline.

4. 输出质量与批处理

作者证明，随着批大小的增加，输出质量并不一定会下降。他们通过在单请求规模下实现GSM8K基准测试74–76%的准确率验证了这一点，并确立了在大规模下维持该性能所需的结构假设。

4. Quality and Batching

The authors demonstrate that output quality does not necessarily degrade with increased batch size. They validated this by achieving 74–76% accuracy on the GSM8K benchmark at the single-request scale and established the structural assumptions required to maintain this performance at scale.

面向未来系统的设计原则 (Design Principles for Future Systems)

该研究得出结论：服务扩散语言模型需要实现从基于AR架构的范式转变： * 步级并行（Step-Level Parallelism）： 系统必须设计为能够在每个去噪步骤的粒度上处理并行性。 * 准入/驱逐逻辑（Admission/Eviction Logic）： 请求准入/驱逐与共享前向传递（Shared Forward Passes）之间的交互与AR模型存在根本不同，这需要全新的调度算法。 * 批处理超时规则（Batch-Timeout Rules）： 作者针对泊松到达条件下的固定填充同步批处理（Fixed-Fill Synchronized Batching），专门推导并提供了一套批处理超时规则。

Design Principles for Future Systems

The study concludes that serving diffusion language models requires a paradigm shift from AR-based architectures: * Step-Level Parallelism: Systems must be designed to handle parallelism at the level of each denoising step. * Admission/Eviction Logic: The interaction between request admission/eviction and shared forward passes differs fundamentally from AR models, necessitating new scheduling algorithms. * Batch-Timeout Rules: The authors provide a derived batch-timeout rule specifically for fixed-fill synchronized batching under Poisson arrival conditions.