跳转至

GAUSS：面向大语言模型推理服务的负载感知与蒙特卡洛仿真器

arXiv:2610.37989 [cs.AI]
Authors: Srikant Bharadwaj, Ankur Mallick, Spyridon Mastorakis, Anjaly Parayil, Dhruv Deshmukh, Victor Ruehle
Submitted: October 27, 2026
Subjects: Artificial Intelligence (cs.AI)
DOI: 10.48550/arXiv.2610.37989

文章背景与核心概要

随着大语言模型 (Large Language Model, LLM) 深度融入生产级应用，如何在保障严格服务等级目标 (SLO) 的同时降低昂贵的算力成本，成为基础设施架构的核心挑战。传统的推理评估往往依赖单一的吞吐量指标，无法捕捉生成式两阶段（预填充与解码）耦合及持续批处理机制所引发的非线性长尾延迟。为此，研究团队推出了名为 GAUSS 的新型负载感知分析与蒙特卡洛仿真器。它通过为预填充和解码构建专属延迟曲面，结合 Lindley 递归排队理论与集成蒙特卡洛采样，实现了对首字延迟 (Time-to-First-Token, TTFT) 与字间延迟 (Time-Between-Tokens, TBT) 全性能分布的精确预测。该工作为未部署硬件评估、新型请求调度算法验证以及精准容量规划提供了兼顾精度与效率的仿真基准。

📌 内容概要

📌 Summary

当前，大语言模型 (LLM) 推理已演进为一项核心的企业级生产负载，需要针对延迟服务等级目标 (SLO) 进行严谨的资源供给、性能监控以及用量计费。然而，传统的单一吞吐量指标在这一场景下往往失效——这是因为生成式服务将预填充 (prefill) 与解码 (decode) 这两个特征截然不同的阶段紧密耦合，而持续批处理 (continuous batching) 机制更是从根本上重构了实际的服务处理流程。

Large language model (LLM) inference has evolved into a core operational workload requiring strict provisioning, monitoring, and billing against latency Service-Level Objectives (SLOs). Traditional single-throughput metrics fall short because generative serving couples two distinct phases (prefill and decode) with vastly different characteristics, while continuous batching fundamentally alters the effective service process.

针对这些局限性，作者团队推出了 GAUSS——一个兼具工作负载感知能力的解析与蒙特卡洛 (Monte-Carlo) 仿真器。该仿真器旨在针对特定工作负载、推理引擎与硬件加速器，精准预测完整的性能分布——尤其是首字延迟 (Time-to-First-Token, TTFT) 与字间延迟 (Time-Between-Tokens, TBT)，并给出系统可稳定承载的可持续请求速率。

To address these limitations, the authors introduce GAUSS, a workload-aware analytical and Monte-Carlo simulator designed to predict full performance distributions—specifically Time-to-First-Token (TTFT) and Time-Between-Tokens (TBT)—alongside sustainable request rates for specific workloads, engines, and accelerators.

🔑 核心特性与方法论

🔑 Key Features & Methodology

  • 阶段专属延迟曲面 (Phase-Specific Latency Surfaces)：为预填充和解码阶段分别配备校准后的专属延迟曲面模型。
  • Lindley 递归排队模型 (Lindley-Recurrence Queueing Model)：精准建模随机请求到达场景下的排队动态与等待行为。
  • 集成蒙特卡洛采样 (Ensemble Monte-Carlo Sampling)：借助高鲁棒性的蒙特卡洛采样而非单纯重放孤立的历史追踪日志来评估系统性能，从而赋能运维团队：
  • 对尚未部署的新型硬件、全新路由策略及假设性工作负载进行前瞻性建模；
  • 精准捕捉传统追踪驱动型仿真器往往采样不足的长尾延迟 (Latency Tails)。
  • 高级容量规划 (Advanced Capacity Planning)：强力支撑关键的运维决策，包括最优工作点选取、流量智能分流，以及基于最短剩余处理时间 (SRPT) 的高效调度策略。
  • Phase-Specific Latency Surfaces: Pairs calibrated latency models for prefill and decode phases.
  • Lindley-Recurrence Queueing Model: Accurately models queueing behavior under stochastic request arrivals.
  • Ensemble Monte-Carlo Sampling: Estimates performance through robust sampling rather than replaying isolated historical traces, enabling operators to:
  • Model undeployed hardware, novel routing policies, and hypothetical workloads.
  • Accurately capture latency tails that trace-driven simulators typically under-sample.
  • Advanced Capacity Planning: Supports critical operational decisions including operating-point selection, traffic segmentation, and shortest-remaining-processing-time (SRPT) scheduling.

🔗 快速链接与资源