跳转至

直接问工具，别瞎猜：智能体工具调用自带运行进度，推理系统理应主动读取

Ask the Tool, Don't Guess: Agent Tool Calls Hold Their Progress, and the Serving System Should Read It

arXiv: 2609.18849 [cs.DC]
Authors: Yipeng Liu, Yingqiang Zhang, Feifei Li, Huanchen Zhang
Submitted: September 16, 2026
Subjects: Distributed, Parallel, and Cluster Computing (cs.DC); Artificial Intelligence (cs.AI); Operating Systems (cs.OS)

文章背景与核心概要

在基于大语言模型 (Large Language Model, LLM) 的 AI 智能体 (AI Agent) 工作流中，调用外部工具往往需要耗费大量物理等待时间，在此期间请求的键值缓存 (KV Cache) 会持续占用极其宝贵的 GPU 显存。当前的大模型推理服务系统大多依赖“盲目猜测”来管理这些缓存（例如仅凭工具名称、历史耗时或调用前预估来决定是否换出显存），但这类静态估算在多变的真实场景下根本无法准确预测工具的运行时间。

针对这一瓶颈，清华大学与合作团队提出了一项全新设计理念：正在运行的工具本身其实掌握着精确的内部进度，只是这些信号此前被智能体技术栈层层屏蔽了。作者团队提出让工具调用在运行时主动向推理系统报告进度，并设计了一套轻量级捕获机制。实验表明，该方案在 KV Cache 置换决策点的准确度相比现有最优预测器提升了数倍乃至一个数量级，在生产级推理引擎中成功将工具调用后的 p90 首字延迟 (Time-to-First-Token, TTFT) 降低超过 20%，性能逼近理想全知调度器 (Oracle)。

总结

Summary

AI 智能体请求在执行过程中，往往需要花费大量的物理时间等待外部工具完成计算。在此期间，该请求对应的键值缓存 (KV Cache) 只能无所事事地滞留在寸土寸金的 GPU 显存当中。目前主流的大语言模型推理系统主要靠“盲猜”来决定如何管理这部分缓存——例如根据工具名称、历史执行耗时、调用前声明的预估时长，或是推理引擎自身的负载情况来进行判断。然而，任何在工具开始运行前就固定下来的静态估算，从根本上都无法准确预测工具的实际耗时，甚至连不同工具调用的耗时先后顺序都排不准。

Agentic requests spend substantial wall-clock time waiting for external tools to execute, during which their KV cache unnecessarily occupies valuable GPU memory. Current LLM serving systems rely on guesswork to manage this cache (such as evaluating tool names, historical execution times, pre-call duration declarations, or engine occupancy), but pre-call estimates fundamentally fail to accurately predict or even rank tool durations.

这篇论文揭示了一个被大家忽视的真相：正在运行的工具自身其实已经掌握了必要的进度数据，只是这些关键信息被智能体软件栈连同工具封装给无声地“静音”了。为此，作者团队提出了一种全新的协作机制，让工具调用在执行过程中能够主动、明确地对外汇报其实时运行进度。

This paper reveals that running tools already possess the necessary progress data, but are silenced by the agent stack. The authors propose a mechanism where tool calls explicitly report their progress during execution.

核心发现与研究成果

Key Findings & Results:

  • 清晰可读的进度信号：在对四个公开智能体语料库进行深入分析后发现，绝大多数工具在运行时都会产生清晰可读取的进度信号——其表现形式要么是剩余工作量的具体百分比，要么是任务即将完工的明确信号。
  • 极低成本的信号捕获：设计了一套轻量级的捕获框架 (Harness)，在完全不改动智能体所见上下文的前提下即可精准提取该信号，对智能体在各项基准测试 (Benchmark) 上的得分没有任何可察觉的负面影响。
  • 远超以往的预测精度：在推理系统决定 KV Cache 留存与置换的关键时刻，工具直接汇报的实时进度比目前文献中最先进的预测器精准数倍乃至一个数量级，并且在运行环境发生波动时依然保持高鲁棒性。
  • 显著的系统性能收益：仅需向生产级推理引擎注入几条轻量级提示信息，相比传统的最近最少使用 (LRU) 缓存淘汰策略，就能将工具调用后的 p90 首字延迟 (Time-to-First-Token, TTFT) 降低 20.7% (仅使用高带宽显存 HBM) 和 20.8% (结合 HBM 与系统内存 DRAM)，效率无限逼近全知理想状态 (Oracle)。
  • Readable Progress Signal: Analyzing four public agent corpora reveals that most tool runtimes contain a clear, readable signal—either as a fraction of remaining work or an indicator that completion is near.
  • Low-Cost Recovery: A lightweight harness recovers this signal without altering what the agent sees, with no measurable impact on benchmark scores.
  • Superior Accuracy: At decision points for KV cache management, reported progress is several times to an order of magnitude more accurate than the best published predictors, remaining resilient to environmental changes.
  • Performance Gains: Integrated into a production serving engine via minimal hints, it cuts the p90 time-to-first-token (TTFT) after a tool call by 20.7% (HBM only) and 20.8% (HBM + DRAM) compared to LRU policies, approaching oracle-level efficiency.

摘要

Abstract

智能体请求在执行时需要耗费大量物理时间等待外部工具，而在此全过程中，其键值缓存 (KV Cache) 一直死死霸占着 GPU 显存。当前的推理服务系统在决定是将这部分缓存保留在显存、换出到内存还是重新加载时，只能依赖对工具运行耗时的盲目猜测——这些猜测往往来自工具名称、历史统计、调用前声明的执行时长，或是推理引擎自身的显存占用率。我们证明了：任何在调用开始前就固化下来的估算手段，在本质上都无法获知其真实的运行时长，甚至连不同工具调用的先后排序都无法正确给出。与此形成鲜明对比的是，正在运行的工具本身早已知晓确切答案，只是智能体软件栈与工具的封装机制一同将这一信号抹杀了。我们提出让工具调用在运行时显式上报其进度，并量化评估了实现这一机制所需的系统开销。对四个公开智能体语料库的普查分析表明，一旦揭开这层遮蔽，绝大部分工具运行时间内都存在清晰可读的信号，其强度主要体现为两类：剩余工作量的百分比，或是即将执行完毕的高置信度信号。我们设计了一套捕获框架，在不改变智能体可见视图的前提下将这些信号完整提取出来，且对智能体的基准测试得分没有任何可测量的损耗。在推理系统做出 KV Cache 置换决策的关键节点，由工具主动汇报的进度其精准度比目前已发表的最优预测器高出数倍乃至一个数量级，即使运行环境发生动态变化也能始终保持精准。将该方案通过少量轻量提示信息接入生产级推理引擎后，相比于经典的 LRU 策略，工具调用后的 p90 首字延迟 (TTFT) 在纯高带宽显存 (HBM) 配置下降低了 20.7%，在 HBM 加系统内存 (DRAM) 混合分层配置下降低了 20.8%，整体表现已极为接近全知理想状态。大模型推理系统完全不必去盲猜那些工具自身就能讲清楚的事情。

An agentic request spends substantial wall-clock time waiting for tools, and its KV cache holds GPU memory the whole time. Serving systems decide whether that cache stays, leaves, or comes back by guessing how long the tool will run, from the tool's name, its history, a duration declared before the call, or the engine's own occupancy. We show that no estimate fixed before a call starts can know its duration, and such estimates may not even rank the calls. Meanwhile, the running tool already holds the answer, but the agent stack together with the tool silences it. We propose that tool calls report their progress explicitly while they run, and we measure what that takes. A census of four public agent corpora finds a readable signal in most tool time once it is revealed, in two strengths: a fraction of the work remaining, or an accurate signal that the end is near. A harness recovers it without changing what the agent sees, at no measurable cost to the agent's benchmark score. At the points where a KV cache decision is made, the reported progress is between several times and an order of magnitude more accurate than the best published predictors, and it stays accurate when the environment changes. Plugged into a production engine through a few small hints, it cuts the p90 time to first token (TTFT) after a tool call by 20.7% (HBM only) and 20.8% (HBM + DRAM) against LRU, close to an oracle. A serving system should not guess what its tools can tell it.

链接与资源