跳转至

Kalypso：关系型大模型推理服务系统

文章背景与核心概要

随着大语言模型（LLM）在非结构化数据处理中的广泛应用，将其作为语义算子（如过滤、提取、排序、连接和转换）进行复杂查询已成为趋势。然而，传统的语义查询处理系统严重依赖以请求为中心（Request-centric）的LLM推理架构，这种架构对整体查询计划一无所知，从而错失了大量性能优化的机会。

本文介绍了 Kalypso，这是一个新型的关系型大模型推理服务系统（Relational LLM Serving）。通过将查询结构感知能力与推理引擎相融合，Kalypso 实现了语义算子之间的流水线式执行（Pipelined Execution），使中间元组能够在算子之间直接流动，从而实现 KV 缓存（KV-cache）状态的高效重用，避免了冗余的重复计算。此外，该系统引入了一种自适应、感知内存的调度算法，能够在运行过程中动态平衡上游并行度、下游进度与 GPU 利用率。评估表明，与传统的以请求为中心的基准系统相比，Kalypso 可将查询完成时间加速高达 4.57 倍。

Kalypso is a novel relational LLM serving system designed to optimize semantic query execution (such as filtering, extracting, ranking, joining, and transforming unstructured data). Traditional semantic query processing systems rely on request-centric LLM serving, which remains blind to the overall query plan and leaves performance optimization opportunities unexploited.

Kalypso introduces relational LLM serving, an abstraction that integrates query structure awareness with the serving engine. By enabling pipelined execution across semantic operators—where intermediate tuples flow directly between operators—Kalypso reuses KV-cache states rather than redundantly recomputing them. Utilizing an adaptive, memory-aware scheduling algorithm, the system continuously balances upstream parallelism, downstream progress, and GPU utilization. Evaluations demonstrate that Kalypso accelerates query completion times by up to 4.57x compared to conventional request-centric baselines.

文档元数据

Document Metadata

字段 (Field) 详情 (Details)
arXiv ID arXiv:2607.23815 [cs.DB]
主题 (Subjects) 数据库 (cs.DB)；人工智能 (cs.AI)；计算与语言 (cs.CL)
作者 (Authors) Hojae Son, Md Ashraful Islam, Huy Gia Cao, Hui Guan, Marco Serafini
提交历史 (Submission History) • v1: 2026年7月26日
• v2 (当前版本): 2026年8月14日
DOI 10.48550/arXiv.2607.23815

摘要

大语言模型正日益被用作处理非结构化数据的语义算子，涵盖过滤、提取、排序、连接和转换等操作。现有的语义查询系统多采用以请求为中心的 LLM 推理架构，对查询计划缺乏感知，导致大量性能优化潜力未能被挖掘。本文提出了“关系型大模型推理服务”（Relational LLM Serving）这一抽象概念，使 LLM 推理服务能够感知语义查询结构，同时确保查询语义与输出准确性不受影响。其核心契机在于语义算子间的流水线执行：当中期元组直接从一个算子流向下一个算子时，可以复用其 KV 缓存状态，而无需重新计算。

本文推出了 Kalypso，这是一个提供语义查询计划 API 的关系型 LLM 推理系统，并采用自适应、感知内存的调度算法对其进行执行。Kalypso 解决了一个全新的在线调度问题：将流水线算子执行与 GPU 内存压力管理相结合，以便在缓存被驱逐之前在推理引擎中重用 KV 缓存状态。其调度器不断调整内存分配，以平衡上游并行度、下游进度和 GPU 利用率。评估结果表明，在各种工作负载下，与基于请求中心 LLM 推理的基准相比，Kalypso 显著提升了查询完成时间，加速比高达 4.57 倍，证明了查询感知型 LLM 推理能够大幅提高语义查询执行的效率。

Large language models are increasingly used as semantic operators for filtering, extracting, ranking, joining, and transforming unstructured data. Existing semantic query processing systems invoke request-centric LLM serving systems that are unaware of the query plan, leaving substantial performance opportunities unused. This paper introduces relational LLM serving, an abstraction that makes LLM serving aware of semantic query structure while preserving query semantics and output accuracy. The key opportunity is pipelined execution across semantic operators: when intermediate tuples flow directly from one operator to the next, their KV-cache state can be reused instead of recomputed.

We present Kalypso, a relational LLM serving system that exposes an API for semantic query plans and executes them using an adaptive, memory-aware scheduling algorithm. Kalypso addresses a new online scheduling problem in which pipelined operator execution is coupled with GPU memory pressure management to reuse KV-cache state in the serving engine before eviction. Its scheduler continuously adjusts memory allocations to balance upstream parallelism, downstream progress, and GPU utilization. Our evaluation shows that Kalypso improves query completion time over baselines using request-centric LLM serving, with speedups up to 4.57x across diverse workloads, demonstrating that query-aware LLM serving can substantially improve the efficiency of semantic query execution.

访问与全文链接

Access & Full-Text Links