跳转至

文章背景与核心概要

在使用强化学习（RL）训练大语言模型（LLM）智能体时，模型往往会学习到脆弱且针对特定任务的捷径，而不是稳健的策略。本文题为《智能体强化学习中的技能复用与压缩》（Skill Reuse as Compression in Agentic RL），提出了一个核心假设：当智能体的成功轨迹在结构上可被压缩为一小组合适的可复用抽象模式时，它们能够实现更好的泛化。

为了将这一思想付诸实践，作者引入了 ReuseRL 方法，该方法将智能体强化学习建立在最小描述长度（MDL）原则的基础之上。ReuseRL 从成功轨迹中提取共享的技能字典，并将分割成本（segmentation cost）引入强化学习目标函数中，从而惩罚那些编码不良的特异性行为。作者证明了一个 PAC-Bayes 边界，保证了未来成功行为具有有界的期望描述长度。在 ALFWorld、TextWorld-Cooking 和 Countdown-Stepwise 等基准测试中，ReuseRL 显著提升了分布内与分布外的成功率。

Skill Reuse as Compression in Agentic RL

Summary

Large language model (LLM) agents trained using reinforcement learning (RL) frequently learn brittle, task-specific shortcuts instead of robust strategies. This paper, Skill Reuse as Compression in Agentic RL, hypothesizes that agents achieve better generalization when their successful trajectories are structurally compressible into a small set of reusable abstract patterns. To operationalize this, the authors introduce ReuseRL, an approach that grounds agentic RL in the Minimum Description Length (MDL) principle. ReuseRL extracts a shared skill dictionary from successful trajectories and incorporates a segmentation cost into the RL objective to penalize idiosyncratic, poorly encoded behaviors. The authors prove a PAC-Bayes bound guaranteeing bounded expected description length for future successful behaviors, and demonstrate that ReuseRL improves both in-distribution and out-of-distribution success across benchmarks like ALFWorld, TextWorld-Cooking, and Countdown-Stepwise.

Paper Metadata

论文元数据

  • arXiv ID: 2605.31509 [cs.LG]
  • Primary Subject: Machine Learning (cs.LG), with cross-listing in Artificial Intelligence (cs.AI)
    • Primary Subject: Machine Learning (cs.LG), with cross-listing in Artificial Intelligence (cs.AI)
  • Authors: Zhikun Xu, Yu Feng, Jacob Dineen, Taiwei Shi, Jieyu Zhao, Ben Zhou
    • Authors: Zhikun Xu, Yu Feng, Jacob Dineen, Taiwei Shi, Jieyu Zhao, Ben Zhou
  • Venue: Accepted by EMNLP 2026 Main Conference
    • Venue: Accepted by EMNLP 2026 Main Conference
  • Submission History:
  • [v1] Fri, 29 May 2026
  • [v2] Tue, 1 Sep 2026 (current version)
    • Submission History:
    • [v1] Fri, 29 May 2026
    • [v2] Tue, 1 Sep 2026 (current version)
  • DOI: 10.48550/arXiv.2605.31509

Abstract

摘要

使用强化学习（RL）训练的大语言模型智能体往往会学习到脆弱的、针对特定任务的捷径。我们假设，当智能体的成功轨迹在结构上可压缩、并可被分解为一小组合适的可复用抽象模式时，它们的泛化能力会更好。为了将这一点形式化，我们引入了 ReuseRL，它将智能体强化学习建立在最小描述长度（MDL）原则之上。ReuseRL 从成功轨迹中提取共享的技能字典，并用分割成本（segmentation cost）来增强强化学习目标，显式惩罚编码不良的特异性行为。我们证明了一个 PAC-Bayes 边界，保证了从成功轨迹中提取的字典在未来的成功行为上具有有界的期望描述长度。在 ALFWorld、TextWorld-Cooking 和 Countdown-Stepwise 中，与朴素的 GRPO 和强轮次长度基线相比，ReuseRL 提高了分布内和分布外的成功率。

Abstract Large language model agents trained with reinforcement learning (RL) often learn brittle, task-specific shortcuts. We hypothesize that agents generalize better when their successful trajectories are structurally compressible, decomposed into a small set of reusable abstract patterns. To formalize this, we introduce ReuseRL, which grounds agentic RL in the Minimum Description Length (MDL) principle. ReuseRL extracts a shared skill dictionary from successful trajectories and augments the RL objective with a segmentation cost, explicitly penalizing idiosyncratic behaviors that encode poorly. We prove a PAC-Bayes bound guaranteeing that a dictionary extracted from successful trajectories has bounded expected description length on future successful behavior. Across ALFWorld, TextWorld-Cooking, and Countdown-Stepwise, ReuseRL improves in- and out-of-distribution success over vanilla GRPO and strong round-length baselines.

Access and Resources

访问与资源