跳转至

文章背景与核心概要

大语言模型（LLM）智能体在面向用户的实际部署中，其核心挑战往往不在于解决任务的上限能力，而在于稳定性和边界意识——即在多次重复试验中保持一致的表现，并准确识别请求是否能够（或能否安全地）被满足。为了弥合智能体偶尔能解决的潜力与其在多次试验中始终保持一致的实际表现之间的差距，本文提出了 TRACE 框架。

TRACE（轨迹对比演化）是一种无需修改模型权重、通过模块化且可检索的技能库来提升智能体行为知识的创新框架。该框架通过智能体自进化循环（对比轨迹分析）动态优化技能库，并在部署阶段执行状态条件下的技能编排。实验证明，TRACE 在车载数字助手等严苛场景（CAR-bench 基准）中取得了显著的性能提升，大幅缩小了 \(\text{Pass@3}\) 与 \(\text{Pass}^3\) 之间的鸿沟，并夺得了官方隐藏评估集的榜首。

TRACE: A Self-Evolving Skill Bank for Consistent, Limit-Aware LLM Agents

TRACE: A Self-Evolving Skill Bank for Consistent, Limit-Aware LLM Agents

Summary

Summary

TRACE (TRAjectory-Contrastive Evolution) is a novel framework designed to bridge the reliability gap in Large Language Model (LLM) agents—specifically addressing the discrepancy between what an agent can solve occasionally (potential performance) and what it achieves consistently across repeated trials. Tested primarily within the demanding domain of in-car digital assistants using the CAR-bench benchmark, TRACE operates without modifying model weights. Instead, it utilizes a modular, retrievable Skill Bank that encodes tool-use rules and behavioral guidelines, dynamically evolving via an agentic self-evolution loop (contrastive trajectory analysis) and state-conditioned skill orchestration.

TRACE (TRAjectory-Contrastive Evolution) is a novel framework designed to bridge the reliability gap in Large Language Model (LLM) agents—specifically addressing the discrepancy between what an agent can solve occasionally (potential performance) and what it achieves consistently across repeated trials. Tested primarily within the demanding domain of in-car digital assistants using the CAR-bench benchmark, TRACE operates without modifying model weights. Instead, it utilizes a modular, retrievable Skill Bank that encodes tool-use rules and behavioral guidelines, dynamically evolving via an agentic self-evolution loop (contrastive trajectory analysis) and state-conditioned skill orchestration.

Key Highlights:

  • Substantial Consistency Gains: On GPT-5.5, TRACE improved consistency (\(\text{Pass}^3\)) by 34.6 points (from 59.9% to 94.5%) and reduced the performance gap to just 4.0 points.
  • Top-Tier Performance: On the official hidden evaluation set, TRACE achieved 1st place using GPT-5.6-Sol, scoring 70% in \(\text{Pass}^3\) (a 40% relative improvement over baselines).
  • Project Resources: Project Homepage

Key Highlights:

  • Substantial Consistency Gains: On GPT-5.5, TRACE improved consistency (\(\text{Pass}^3\)) by 34.6 points (from 59.9% to 94.5%) and reduced the performance gap to just 4.0 points.
  • Top-Tier Performance: On the official hidden evaluation set, TRACE achieved 1st place using GPT-5.6-Sol, scoring 70% in \(\text{Pass}^3\) (a 40% relative improvement over baselines).
  • Project Resources: Project Homepage

Metadata

Metadata

  • arXiv ID: arXiv:2608.22793 [cs.CL]
  • Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
  • Authors: Wenhao Wu, Menghao Zhang, Xin Wang, Zhi Wang, Kun Shao, Jian Luan
  • Submitted: August 24, 2026
  • Citation: Wu, W., Zhang, M., Wang, X., Wang, Z., Shao, K., & Luan, J. (2026). TRACE: A Self-Evolving Skill Bank for Consistent, Limit-Aware LLM Agents. arXiv:2608.22793.
  • arXiv ID: arXiv:2608.22793 [cs.CL]
  • Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
  • Authors: Wenhao Wu, Menghao Zhang, Xin Wang, Zhi Wang, Kun Shao, Jian Luan
  • Submitted: August 24, 2026
  • Citation: Wu, W., Zhang, M., Wang, X., Wang, Z., Shao, K., & Luan, J. (2026). TRACE: A Self-Evolving Skill Bank for Consistent, Limit-Aware LLM Agents. arXiv:2608.22793.

Abstract

Abstract

Reliable deployment of LLM agents in user-facing products depends not on raw task-solving ability but on consistency and limit-awareness: behaving the same way across repeated trials, and recognizing when a request cannot, or cannot yet, be safely fulfilled. CAR-bench exposes this reliability gap in the domain of in-car assistants: an LLM-simulated user issues incomplete or ambiguous requests, requiring the agent to resolve uncertainty through multi-turn dialogue and tool use while strictly adhering to domain policies.

Reliable deployment of LLM agents in user-facing products depends not on raw task-solving ability but on consistency and limit-awareness: behaving the same way across repeated trials, and recognizing when a request cannot, or cannot yet, be safely fulfilled. CAR-bench exposes this reliability gap in the domain of in-car assistants: an LLM-simulated user issues incomplete or ambiguous requests, requiring the agent to resolve uncertainty through multi-turn dialogue and tool use while strictly adhering to domain policies.

Even frontier models show a substantial gap between what they can solve at least once (\(\text{Pass@3}\)) and what they solve consistently across trials (\(\text{Pass}^3\)). We bridge this gap with TRACE, which iteratively improves a skill-based agent's behavioral knowledge without modifying model weights. This knowledge is organized as a Skill Bank of modular, retrievable skills, each encoding a self-contained set of tool-use rules and behavioral guidelines. TRACE evolves this bank through an agentic self-evolution loop: after each evaluation round, it groups trajectories by the skills invoked and refines each skill by contrasting successful and failed behaviors. The updated bank then guides subsequent rounds, while during deployment the Actor performs state-conditioned skill orchestration at every turn.

Even frontier models show a substantial gap between what they can solve at least once (\(\text{Pass@3}\)) and what they solve consistently across trials (\(\text{Pass}^3\)). We bridge this gap with TRACE, which iteratively improves a skill-based agent's behavioral knowledge without modifying model weights. This knowledge is organized as a Skill Bank of modular, retrievable skills, each encoding a self-contained set of tool-use rules and behavioral guidelines. TRACE evolves this bank through an agentic self-evolution loop: after each evaluation round, it groups trajectories by the skills invoked and refines each skill by contrasting successful and failed behaviors. The updated bank then guides subsequent rounds, while during deployment the Actor performs state-conditioned skill orchestration at every turn.