文章背景与核心概要
在大语言模型（LLM）驱动的智能体强化学习领域，基于结果的强化学习方法（如 GRPO）虽然能够帮助智能体解决复杂的长周期任务，但交互轨迹中蕴含的宝贵探索模式通常在单次策略更新后就被丢弃了。现有的增强经验方法往往依赖于推理时的外部检索，这不仅导致了持续的外部依赖，还无法适应策略不断演进的能力。
为了解决这一痛点，本文提出了 EDGE（Experience-Distillation for Guided Exploration）框架。该框架将检索到的经验视为“临时的训练期脚手架”，通过经验蒸馏逐步将这些经验的优势内化到参数化策略中。具体而言，EDGE 通过划分经验条件轨迹与无经验轨迹来评估并采纳正向边际收益，利用反向 KL 目标将引导行为蒸馏至基础策略，并通过协同演进的经验库动态合成指导意见并剔除过时条目。实验表明，该方法在 ALFWorld 和 WebShop 基准测试中展现出显著的性能提升。
EDGE: Experience-Distillation for Guided Exploration in Agentic Reinforcement Learning
EDGE: Experience-Distillation for Guided Exploration in Agentic Reinforcement Learning
Overview & Summary
Overview & Summary
EDGE (Experience-Distillation for Guided Exploration) is a novel framework designed to enhance agentic reinforcement learning for Large Language Models (LLMs). While outcome-based reinforcement learning techniques (such as GRPO) help LLM agents solve complex, long-horizon tasks, valuable exploration patterns found within interaction trajectories are usually discarded after a single policy update.
EDGE (Experience-Distillation for Guided Exploration) is a novel framework designed to enhance agentic reinforcement learning for Large Language Models (LLMs). While outcome-based reinforcement learning techniques (such as GRPO) help LLM agents solve complex, long-horizon tasks, valuable exploration patterns found within interaction trajectories are usually discarded after a single policy update.
Existing experience-augmented approaches rely on external retrieval at inference time—leading to persistent dependencies and a failure to adapt to the policy's evolving capabilities. EDGE solves this by treating retrieved experiences as temporary training-time scaffolds, progressively internalizing their benefits directly into the parametric policy via experience distillation.
Existing experience-augmented approaches rely on external retrieval at inference time—leading to persistent dependencies and a failure to adapt to the policy's evolving capabilities. EDGE solves this by treating retrieved experiences as temporary training-time scaffolds, progressively internalizing their benefits directly into the parametric policy via experience distillation.
Key Paper Details
Key Paper Details
- arXiv Identifier: arXiv:2608.21946 [cs.CL]
- Accepted Venue: EMNLP 2026 (Main Conference)
- Primary Subject: Computation and Language (
cs.CL) - Other Subjects: Artificial Intelligence (
cs.AI), Machine Learning (cs.LG) - Submission Date: August 22, 2026
- Authors: Can Xie, Yuyi Zhou, Wen Yang, Ziyi zhang, Siyao Song, Yingzhuo Deng, Shuo Ren, and Jiajun Zhang
- Official Repository: GitHub - xvolcano02/EDGE
- arXiv Identifier: arXiv:2608.21946 [cs.CL]
- Accepted Venue: EMNLP 2026 (Main Conference)
- Primary Subject: Computation and Language (
cs.CL)- Other Subjects: Artificial Intelligence (
cs.AI), Machine Learning (cs.LG)- Submission Date: August 22, 2026
- Authors: Can Xie, Yuyi Zhou, Wen Yang, Ziyi zhang, Siyao Song, Yingzhuo Deng, Shuo Ren, and Jiajun Zhang
- Official Repository: GitHub - xvolcano02/EDGE
Abstract
Abstract
Reinforcement learning with outcome-based objectives such as GRPO enables LLM-based agents to solve complex, long-horizon tasks, yet the reusable exploration patterns embedded in interaction trajectories are largely discarded after a single policy update. Existing experience-augmented approaches retrieve historical guidance at inference time, but they apply experiences without accounting for the policy's evolving capability and create persistent dependencies on external retrieval. We propose EDGE (Experience-Distillation for Guided Exploration), a framework that treats retrieved experiences as temporary training-time scaffolds and progressively internalizes their benefits into the parametric policy. Concretely, EDGE partitions each rollout group into experience-conditioned and experience-free trajectories to estimate and admit only positive marginal gains without extra sampling, then distills the induced behavior into the base policy via a reverse-KL objective on its own empirical support. A co-evolutionary experience bank further synthesizes guidance from emerging failure modes and prunes obsolete entries as the policy evolves. On ALFWorld and WebShop, EDGE improves over GRPO by 8.3 and 12.5 success-rate points at the 7B scale and retains 96.0% of its scaffolded performance when external experiences are removed at inference time.
Core Methodology & Highlights
Core Methodology & Highlights
- Temporary Training-Time Scaffolding: Instead of forcing persistent external retrieval at inference time, EDGE internalizes external guidance directly into the base policy's parameters.
- Marginal Gain Estimation: Partitions rollout groups into experience-conditioned and experience-free trajectories to identify and admit positive marginal gains efficiently without requiring extra sampling.
- Reverse-KL Distillation: Distills induced behaviors into the base policy using a reverse-KL objective built on its own empirical support.
- Co-Evolutionary Experience Bank: Dynamically synthesizes guidance tailored to emerging failure modes while simultaneously pruning obsolete entries as the policy continues to evolve.
- Empirical Performance:
- Outperforms standard GRPO by 8.3 (ALFWorld) and 12.5 (WebShop) success-rate points at the 7B parameter scale.
- Retains 96.0% of its scaffolded performance even when external experience modules are entirely removed during inference.
- Temporary Training-Time Scaffolding: Instead of forcing persistent external retrieval at inference time, EDGE internalizes external guidance directly into the base policy's parameters.
- Marginal Gain Estimation: Partitions rollout groups into experience-conditioned and experience-free trajectories to identify and admit positive marginal gains efficiently without requiring extra sampling.
- Reverse-KL Distillation: Distills induced behaviors into the base policy using a reverse-KL objective built on its own empirical support.
- Co-Evolutionary Experience Bank: Dynamically synthesizes guidance tailored to emerging failure modes while simultaneously pruning obsolete entries as the policy continues to evolve.
- Empirical Performance:
- Outperforms standard GRPO by 8.3 (ALFWorld) and 12.5 (WebShop) success-rate points at the 7B parameter scale.
- Retains 96.0% of its scaffolded performance even when external experience modules are entirely removed during inference.