跳转至

文章背景与核心概要

大语言模型 (Large Language Model, LLM) 智能体 (AI Agent) 正在被寄予厚望去攻克复杂的长程任务，但在漫长的探索过程中，面对岔路口选择哪条技术路线，往往直接决定了整项任务的成败。现有的评估基准普遍只看最终结果是否成功，忽视了智能体在关键转折点的中间判断力——作者将这种在长程任务中做出明智方向决策的能力定义为智能体的“决策品味 (Taste) ”。为填补这一评估空白，研究团队无需任何人工标注，直接从工程与科研任务轨迹中自动挖掘分叉点，构建了全新的自动化基准测试 (Benchmark) Taste-Bench。评测发现即便是当前最强大的前沿模型也难以应对这一挑战 (最高准确率仅为 59.7%) ，且简单增加推理算力无法改善品味；但令人振奋的是，通过蒸馏具有后验视角的教师模型判断力，可以有效培养智能体的品味，并在具挑战性的 SWE-bench Pro 上显著提升了端到端的任务成功率。

有品味的智能体：长程任务中决策品味 Taste 的度量与提升

The Tasteful Agent: Measuring and Improving Taste in Long-Horizon Tasks

arXiv 编号: 2609.25804 [cs.AI]
论文作者: Wenbo Pan, Zhichao Liu, Shujie Liu, Jingying Zeng, Chin-Yew Lin, Xianfeng Tang, Yan Lu, Qi He, Xiaohua Jia
提交时间: 2026年9月22日 (修订于：2026年9月23日)

arXiv: 2609.25804 [cs.AI]
Authors: Wenbo Pan, Zhichao Liu, Shujie Liu, Jingying Zeng, Chin-Yew Lin, Xianfeng Tang, Yan Lu, Qi He, Xiaohua Jia
Submitted: September 22, 2026 (Revised: September 23, 2026)

📌 概要

📌 Summary

随着大语言模型 (Large Language Model, LLM) 智能体 (AI Agent) 面对日益复杂的多步骤长程 ("long-horizon") 任务，它们在探索推进过程中所做出的各项中间决策——例如决定优先检验哪一个科研假设，抑或选择哪种架构方案进行落地构建——在根本上主导了最终的成败。在本文中，作者将智能体在长程探索中审时度势、做出卓越决策的核心能力，生动地定义为智能体的“品味 (Taste) ”。

As Large Language Model (LLM) agents tackle increasingly complex, multi-step ("long-horizon") tasks, their intermediate decisions—such as choosing which hypothesis to test or architectural implementation to build—ultimately dictate their success. The authors define the capability to make superior long-horizon decisions as an agent's "taste."

由于现有的评估基准测试 (Benchmark) 普遍只关注智能体最终是否跑通了端到端任务，而忽视了对其行进过程中的中间判断力进行考量，作者团队提出了 Taste-Bench——一个无需任何人工标注、完全从工程与科研真实任务轨迹中自动化挖掘构建的全新基准测试。评测结果揭示了一个耐人寻味的现象：即便是当前最先进的前沿大模型在面对这项指标时也显得力不从心 (最高准确率仅为 59.7%) ，而且单纯为模型分配更多的推理计算预算也无法真正提升其决策品味。不过令人振奋的是，研究团队证实品味确实能够借助知识蒸馏被有效习得，这不仅显著优化了智能体面对陌生未知任务时的决策水平，更直接带动了在高难度软件工程评测 SWE-bench Pro 上的端到端任务成功率。

Because current benchmarks only evaluate end-to-end task completion rather than intermediate judgment, the authors introduce Taste-Bench, a novel automated benchmark mined from real agent trajectories in engineering and research tasks without human annotation. Their evaluations reveal that even frontier models struggle with this metric (topping out at 59.7% accuracy), and that simply allocating a larger reasoning budget does not improve taste. Finally, they demonstrate that taste can be successfully trained via distillation, improving both decision-making on unseen tasks and end-to-end success on SWE-bench Pro.

📑 概览与元数据

📑 Overview & Metadata

📝 摘要

📝 Abstract

大语言模型智能体在长程复杂任务中的应用正变得日益广泛，而它们在探索路径上所迈出的每一步中间抉择——比如优先验证哪个潜在假设，抑或选择在哪个技术方案的基础之上继续推进——直接决定了整个运行流程的最终结局。能够在这些关键节点上做出精准而富有远见的决策，正在成为工程型与科研型智能体不可或缺的核心竞争力。在本文中，我们将这种在长程任务中明辨方向、做出优异决策的高阶判断力，统称为智能体的“品味 (Taste) ”。

LLM agents increasingly work on long-horizon tasks, and the decisions they make along the way, such as which hypothesis to test or which implementation to build on, determine the outcome of the whole run. Making these decisions well is becoming a key capability for both engineering and research agents. We refer to the ability to make good long-horizon decisions as the taste of an agent.

尽管现有的各类基准测试能够衡量智能体在长程任务中的端到端成功概率，但没有一项基准能够单独度量智能体的品味。针对这一空白，我们打造了 Taste-Bench——这是一个完全由智能体在真实工程与科研任务中生成的运行轨迹自动构建而成的品味评测基准。测试中的每一个问题都对应着一个关键的“决策分叉点 (Decision Fork) ”，即在任务推进的某一特定时刻，前方同时存在数条可行路径，且其中一条已被验证能导向更优的结果；被测模型必须在完全无法窥探分叉后续演变的前提下，仅凭当前上下文在多个方向中做出决断。

While existing benchmarks measure the end-to-end success of agents on long-horizon tasks, none of them measures the taste of an agent. To address this problem, we build Taste-Bench, a benchmark of taste questions constructed automatically from trajectories that agents produced in engineering and research tasks. Each question presents a decision fork, a point in a trajectory where multiple directions are available and one of them leads to a better outcome, and the evaluated model chooses among these directions without seeing what happens after the fork.

我们在无需任何人工标注介入的情况下，通过比对同一任务的多条并行尝试轨迹，并挖掘单次探索中陷入弯路后修正折返的迂回片段，全自动化提取出了这些宝贵的分叉点。我们在 Taste-Bench 上系统评测了多款业界前沿模型，结果发现即便表现最佳的模型，其答题正确率也仅为 59.7%。深入分析表明，若决定某条路径优劣的关键证据出现在轨迹中较靠后的阶段，所有模型的判断难度都会急剧飙升；更关键的是，单纯增加思考时间或扩大推理计算预算，并不能实质性改善模型的判断准确率。最后，我们证实了品味并非不可捉摸，而是完全可以被后天训练的：通过将拥有完整执行后验视角的教师模型判断力蒸馏给学生模型，学生模型在未曾见过的全新任务中展现出了更优越的决策能力，并在严格留出的 SWE-bench Pro 复杂任务评测中大幅刷新了端到端解决成功率。

We mine these forks automatically from parallel attempts at the same task and from detours inside a single trajectory, without needing human annotation. We evaluate frontier models on Taste-Bench and find that the best model answers only 59.7% of the questions correctly. We further find that forks whose deciding evidence appears later in the trajectory are much harder for every model, and that a larger reasoning budget does not improve the accuracy. Finally, we show that taste can be trained. We distill the judgment of a teacher that has seen the outcome into a student model, and the student makes better decisions on unseen tasks and improves end-to-end success on held-out SWE-bench Pro tasks.