跳转至

文章背景与核心概要

在长周期（long-horizon）工具调用场景中，训练大语言模型（LLM）智能体通常依赖于与真实或合成的可执行环境进行交互。然而，这一过程成本高昂、难以验证，且受制于外部模拟器。为了克服这些局限性，作者引入了一种名为 EnvACE 的新型智能体强化学习方法，该方法通过“世界彩排”（world rehearsal）在训练期间取代了对外部环境的交互。策略不再查询外部设置，而是在“行动”（生成工具调用）与“彩排”（扮演环境角色以产生诱导响应）之间交替进行，并将后续决策条件化于这些彩排输出之上。

这两种角色通过任务成功奖励进行端到端的联合优化，使策略能够直接将环境动力学内化到其参数中。在测试时，这种内化的世界模型支持在正式执行前进行私密彩排，在适度预算下提升性能，而无需额外的外部交互。这项研究为突破外部环境约束、扩展LLM智能体训练开辟了一条新路径。

EnvACE: Internalizing Environment Dynamics via World Rehearsal for Agentic Reinforcement Learning

arXiv: 2608.06197 [cs.AI]
Submitted: August 6, 2026
Authors: Zishan Xu, Zhiyuan Yao, Yuxin Chen, Yifu Guo, Zhengxi Lu, Yuquan Lu, Jinyang Huang, Yan Xu, Yasheng Wang, Weinan Zhang, Xingshan Zeng, Weiwen Liu
Repository: GitHub - Within-yao/EnvACE

📌 Executive Summary

Training Large Language Model (LLM) agents for long-horizon tool use typically relies on interacting with real or synthesized executable environments—a process that is costly, hard to verify, and constrained by external simulators.

To overcome these limitations, the authors introduce EnvACE, a novel agentic reinforcement learning method that replaces external environment interactions during training with world rehearsal. Instead of querying an external setup, the policy alternates between: 1. Acting: Generating a tool call. 2. Rehearsing: Playing the role of the environment to produce the induced response, conditioning subsequent decisions on this rehearsed output.

Both roles are jointly optimized end-to-end using task-success rewards, enabling the policy to internalize environment dynamics directly into its parameters. At test time, this internalized world model supports private rehearsal before committed execution, boosting performance under a moderate budget without requiring additional external interactions.

📑 Abstract

训练用于长周期工具调用的大语言模型智能体，通常依赖于与真实或合成的可执行环境进行交互（其构建和验证成本高昂），或是依赖难以落地的外部模拟器。我们引入了 EnvACE，这是一种智能体强化学习方法，它在训练期间用“世界彩排”取代了外部环境交互。该策略在行动和彩排之间交替：它首先生成一个工具调用，然后扮演环境角色来产生由该动作引发的响应，并将后续决策条件化于彩排后的响应。这两种角色利用任务成功奖励进行端到端的联合优化。通过世界彩排，策略将动作与其环境响应之间的关系内化到其参数中，从而产生一个直接支持决策的智能体世界模型。在 BFCL-v4、\(\tau^2\)-Bench、VitaBench 和 FinMCP-Bench 上，EnvACE 取得了强大且可迁移的性能，在整体评估中优于环境扩展基线。对照研究进一步表明，世界彩排在不同模型规模下都能持续改进策略学习。在测试时，内化的世界模型使得在正式执行前进行私密彩排成为可能，在适度的彩排预算下带来进一步的性能提升，且无需额外的外部交互。我们的研究结果确立了世界彩排作为一种突破外部环境约束扩展 LLM 智能体训练的新途径。

Training large language model agents for long-horizon tool use typically relies on interactions with real or synthesized executable environments, whose construction and verification are costly, or on external simulators that are difficult to ground. We introduce EnvACE, an agentic reinforcement learning method that replaces external environment interaction during training with world rehearsal. The policy alternates between acting and rehearsal: it first generates a tool call, then plays the role of the environment to produce the response induced by that action, and conditions subsequent decisions on the rehearsed response. Both roles are jointly optimized end-to-end using task-success rewards. Through world rehearsal, the policy internalizes the relationship between actions and their environment responses in its parameters, yielding an agent world model that directly supports decision making. Across BFCL-v4, tau^2-Bench, VitaBench, and FinMCP-Bench, EnvACE achieves strong and transferable performance, outperforming environment-scaling baselines in the overall evaluation. Controlled studies further show that world rehearsal consistently improves policy learning across model scales. At test time, the internalized world model enables private rehearsal before committed execution, yielding further gains under a moderate rehearsal budget without additional external interaction. Our findings establish world rehearsal as a new path toward scaling LLM agent training beyond the constraints of external environments.

🔬 Key Contributions & Methodology

  • 世界彩排框架（World Rehearsal Framework）： 通过让智能体模拟环境对其自身工具调用的响应，消除了训练期间对外部环境设置的依赖。
  • 端到端联合优化（End-to-End Joint Optimization）： 利用任务成功奖励，同时优化智能体的动作策略与其环境响应生成。
  • 内化世界模型（Internalized World Model）： 将环境动力学直接嵌入模型的参数中，为推理和决策创建了稳健的基础。

  • 测试时私密彩排（Test-Time Private Rehearsal）： 赋能模型在承诺执行之前对工具交互进行内部“心理”预演，在无需额外外部 API 或环境调用的情况下实现性能提升。

  • World Rehearsal Framework: Eliminates the dependency on external environment setups during training by letting the agent simulate the environment's responses to its own tool calls.

  • End-to-End Joint Optimization: Optimizes both the agent's action policy and its environmental response generation simultaneously using task-success rewards.
  • Internalized World Model: Embeds environment dynamics directly into the model's parameters, creating a robust foundation for reasoning and decision-making.
  • Test-Time Private Rehearsal: Empowers the model to perform internal "mental" dry-runs of tool interactions before committing to execution, achieving performance gains without extra external API or environment calls.

📊 Evaluation & Results

EnvACE 在几个要求严苛的智能体基准测试中进行了全面评估： * BFCL-v4 * \(\tau^2\)-Bench * VitaBench * FinMCP-Bench

核心发现： * 在整体评估中优于标准的扩展环境基线（environment-scaling baselines）。 * 在多样化的基准测试中展现出强大的可迁移性。 * 在各种模型规模下均显示出持续的策略学习改进。

EnvACE was comprehensively evaluated across several demanding agent benchmarks: * BFCL-v4 * \(\tau^2\)-Bench * VitaBench * FinMCP-Bench

Key Findings: * Outperforms standard environment-scaling baselines across overall evaluations. * Demonstrates robust transferability across diverse benchmarks. * Shows consistent policy learning improvements across various model scales.