文章背景与核心概要
大语言模型（LLM）工具智能体（Tool-Agent）在无需重新训练模型的情况下，可以通过优化其运行时“外挂框架”（harness）——包括周边的提示词、工具接口、中间件、状态处理以及恢复逻辑——来实现性能的大幅提升。本文被 EMNLP 2026 录用，研究者将此建模为一个针对固定模型多轮工具智能体的资源有界外挂框架选择问题，重点关注提示词和工具边界中间件。
为了可靠地评估这些优化，作者引入了一种与优化器无关的评估协议，能够测量独立测试集提升（held-out lift）、最差条件提升（worst-condition lift）、可重复性，以及一项保守的性能指标（\(\text{RelLift}_{95}(B)\)）。此外，他们还提出了 PRISM，这是一个帕累托搜索优化器（Pareto-search optimizer），能够聚类失败案例并将修复方案路由至适当的修改表面，在多个行业基准测试中展现出显著的性能提升。
Beyond Prompts: Measuring and Optimizing LLM Tool-Agent Harnesses
Beyond Prompts: Measuring and Optimizing LLM Tool-Agent Harnesses
Executive Summary
This paper, accepted at EMNLP 2026, explores how Large Language Model (LLM) tool-agents can be enhanced without retraining by optimizing their runtime "harness"—the surrounding prompts, tool interfaces, middleware, state handling, and recovery logic. The authors model this as a resource-bounded harness selection problem for fixed-model multi-turn tool agents, focusing specifically on prompts and tool-boundary middleware. To evaluate these optimizations reliably, they introduce an optimizer-agnostic protocol measuring held-out lift, worst-condition lift, repeatability, and a conservative performance metric (\(\text{RelLift}_{95}(B)\)). They also propose PRISM, a Pareto-search optimizer that clusters failures and routes repairs to appropriate edit surfaces, demonstrating significant performance gains across multiple industry benchmarks.
Executive Summary
This paper, accepted at EMNLP 2026, explores how Large Language Model (LLM) tool-agents can be enhanced without retraining by optimizing their runtime "harness"—the surrounding prompts, tool interfaces, middleware, state handling, and recovery logic. The authors model this as a resource-bounded harness selection problem for fixed-model multi-turn tool agents, focusing specifically on prompts and tool-boundary middleware. To evaluate these optimizations reliably, they introduce an optimizer-agnostic protocol measuring held-out lift, worst-condition lift, repeatability, and a conservative performance metric (\(\text{RelLift}_{95}(B)\)). They also propose PRISM, a Pareto-search optimizer that clusters failures and routes repairs to appropriate edit surfaces, demonstrating significant performance gains across multiple industry benchmarks.
Metadata
Metadata
- arXiv ID:
arXiv:2609.05736[cs.AI]- arXiv ID:
arXiv:2609.05736[cs.AI]
- arXiv ID:
- Primary Subject: Artificial Intelligence (
cs.AI)- Primary Subject: Artificial Intelligence (
cs.AI)
- Primary Subject: Artificial Intelligence (
- Submission Date: September 4, 2026
- Submission Date: September 4, 2026
- Venue: Accepted to EMNLP 2026 (18 pages, 9 figures, 10 tables)
- Venue: Accepted to EMNLP 2026 (18 pages, 9 figures, 10 tables)
- Authors:
- Cen (Mia) Zhao
- Haibo Ruan
- Wenjie Chen
- Pei-fen Tu
- Usman Abbasi
- Joel Hesch
- Authors:
- Cen (Mia) Zhao
- Haibo Ruan
- Wenjie Chen
- Pei-fen Tu
- Usman Abbasi
- Joel Hesch
Abstract & Overview
现代 LLM 工具智能体可以通过精炼其运行时执行环境，在无需进行昂贵模型重训的情况下获得巨大提升。这项研究并没有允许对智能体执行逻辑进行任意重写，而是将搜索范围严格限制在提示词和工具边界中间件（放置在工具边界处的受保护拦截层）上。
为了确保评估的严谨性，作者引入了一个全面的、与优化器无关的评估协议，报告以下内容： 1. 平均独立测试集提升（Mean held-out lift） 2. 最差条件提升（Worst-condition lift） 3. 可重复性（Repeatability） 4. 记录的成本诊断（Logged cost diagnostics） 5. \(\text{RelLift}_{95}(B)\)：在特定预算 \(B\) 下选择的外挂框架的独立测试集增益的保守估计。
Abstract & Overview
Modern LLM tool-agents can be vastly improved without expensive model retraining by refining their runtime execution environment. Rather than allowing arbitrary rewrites of agent execution logic, this research scopes the search surface strictly to prompts and tool-boundary middleware (guarded intercepts placed at the tool boundary).
To ensure rigorous evaluation, the authors introduce a comprehensive, optimizer-agnostic protocol that reports: 1. Mean held-out lift 2. Worst-condition lift 3. Repeatability 4. Logged cost diagnostics 5. \(\text{RelLift}_{95}(B)\): A conservative estimate of the held-out gain of a harness selected under a specific budget \(B\).
Methodological Highlight: PRISM
该协议通过仅使用提示词以及“提示词+中间件”的优化器进行了实例化，突出了 PRISM 的作用。 * 机制： PRISM 在帕累托搜索框架内，对智能体的失败案例进行聚类，并将修复方案策略性地路由至提示词、工具边界中间件或联合修改表面。 * 主要发现： 在包括 BFCL multi-round、tau2-Retail 和 tau2-Telecom 在内的基准测试集中，PRISM 分别实现了 14.2、14.9 和 10.1 个百分点的显著平均独立测试集提升，且这三个基准测试的实证 \(\text{RelLift}_{95}\) 均为正值。 * 组件消融： 研究将 PRISM 的成功主要归功于其失败表面路由（failure-surface routing）以及严格的修改模式约束（edit-pattern constraints）。
Methodological Highlight: PRISM
The protocol is instantiated using both prompt-only and prompt-plus-middleware optimizers, highlighting PRISM. * Mechanism: PRISM clusters agent failures and strategically routes repairs to prompts, tool-boundary middleware, or joint edit surfaces within a Pareto search framework. * Key Findings: Across benchmark suites including BFCL multi-round, tau2-Retail, and tau2-Telecom, PRISM achieved remarkable mean held-out lifts of 14.2, 14.9, and 10.1 percentage points, respectively, alongside positive empirical \(\text{RelLift}_{95}\) across all three benchmarks. * Component Ablation: The study attributes PRISM's success primarily to its failure-surface routing and strict edit-pattern constraints.
Key Takeaways
- 避免脆弱的更新： 尽管某些搜索过程偶尔能发现巨大的性能峰值，但它们往往会选择脆弱的更新。因此，所选外挂框架的可靠性和可重复性必须始终与平均独立测试集提升一同报告。
- 目标明确的拦截层即足矣： 将修改限制在受保护的工具边界拦截层和提示词修改上，可以为全面模型重训提供一种强大且资源高效的替代方案。
Key Takeaways
- Avoid Brittle Updates: While certain search procedures can occasionally uncover massive performance spikes, they frequently opt for brittle updates. Therefore, the reliability and repeatability of a chosen harness must always be reported alongside average held-out lift.
- Targeted Intercepts Suffice: Constraining edits to guarded tool-boundary intercepts and prompt modifications provides a powerful, resource-efficient alternative to full-scale model retraining.
Full-Text & Resources
- 查看 PDF (View PDF)
- HTML 版本 - 实验性 (HTML Version (Experimental))
- TeX 源码 (TeX Source)
- DOI: 10.48550/arXiv.2609.05736
Full-Text & Resources