跳转至

文章背景与核心概要

随着大语言模型（LLM）推理向多租户 GPU 集群转型，合批处理（co-batching）虽然提升了整体吞吐量，但也掩盖了每个租户的具体资源消耗，从而限制了精细化控制。为了在推理引擎中实现按比例的资源共享，业界亟需一种实时、按请求划分的归因原语——它既要足够精准，又要足够轻量，以便直接在调度循环内部执行。

本文介绍了 LLMVisor，这是一个由 Roofline 模型指导的延迟归因模型。通过在与 FLOPs 和内存 I/O 流量成正比的特征上采用简洁的分段线性形式，LLMVisor 有效捕捉了受内存带宽瓶颈（memory-bound）和计算瓶颈（compute-bound）制约的各个阶段。它能够以微秒级的规模，将批处理延迟分解为可加的、按请求划分的份额。

LLMVisor: A Real-Time Latency Attribution Model for Multi-Tenant LLM Serving

Summary

As Large Language Model (LLM) inference shifts toward multi-tenant GPU clusters, co-batching improves overall throughput but obscures per-tenant usage and limits granular control. To enable fractional resource sharing in inference engines, a real-time, per-request attribution primitive is required—one that is both accurate and lightweight enough to execute directly inside the scheduling loop.

This paper introduces LLMVisor, a roofline-guided latency attribution model. By utilizing a concise piecewise-linear form over features proportional to FLOPs and memory I/O traffic, LLMVisor effectively captures both memory-bound and compute-bound phases. It decomposes batch latency into additive, per-request shares operating at a microsecond scale.

Metadata

Authors

  • Shuowei Jin
  • Xueshen Liu
  • Jiaxin Shan
  • Le Xu
  • Tieying Zhang
  • Liguang Xie
  • Z. Morley Mao

Abstract

As LLM inference shifts to multi-tenant GPU clusters, co-batching throughput but obscures per-tenant usage and limits control. Enabling fractional sharing of the inference engine requires a real-time, per-request attribution primitive that is accurate and light enough to run inside the scheduling loop.

We present LLMVisor, a roofline-guided latency attribution model that captures the memory-bound and compute-bound phases via a concise piecewise-linear form over features proportional to FLOPs and memory I/O traffic. LLMVisor decomposes batch latency into additive, per-request shares and runs efficiently at microsecond scale.

We evaluate LLMVisor across Llama 3.1-8B and Qwen 2.5-14B/32B on A100/H100 GPUs under varying tensor parallelism and workload mixes. Compared to a token-count baseline, LLMVisor attains near-perfect \(R^2\) and reduces relative error by up to 2.5x and 3.3x at p90 and p99, respectively, for prefill, and by up to 3.5x and 4.4x for decode, despite batching variability and sequence divergence.

核心亮点与性能表现

  • Roofline 导向设计： 利用与 FLOPs 和内存 I/O 流量成正比的指标，精准建模内存带宽受限和计算受限的特征。
  • 微秒级开销： 运行效率高，可在活跃的调度循环中执行，不会引入明显的延迟瓶颈。
  • 稳健的评估： 在工业级标准模型（Llama 3.1-8B、Qwen 2.5-14B/32B）以及 A100 和 H100 GPU 上进行了广泛测试。
  • 卓越的准确性： 显著优于简单的 Token 计数基线方法，实现了：
  • 在预填充（prefill）阶段，p90/p99 相对误差分别降低高达 2.5倍 / 3.3倍。
  • 在解码（decode）阶段，p90/p99 相对误差分别降低高达 3.5倍 / 4.4倍。

Key Highlights & Performance

  • Roofline-Guided Design: Accurately models memory-bound and compute-bound characteristics using metrics proportional to FLOPs and memory I/O traffic.
  • Microsecond-Scale Overhead: Efficient enough to run within the active scheduling loop without adding noticeable latency bottlenecks.
  • Robust Evaluation: Tested extensively on industry-standard models (Llama 3.1-8B, Qwen 2.5-14B/32B) across A100 and H100 GPUs.
  • Superior Accuracy: Significantly outperforms naive token-count baselines, achieving:
  • Up to 2.5x / 3.3x reduction in relative error at p90/p99 for the prefill phase.
  • Up to 3.5x / 4.4x reduction in relative error at p90/p99 for the decode phase.

访问与链接