跳转至

文章背景与核心概要

随着大语言模型在软件工程领域的广泛应用，如何高效训练具备复杂工具调用和自主排错能力的“代码智能体（Coding Agents）”成为了当前人工智能研究的热点与难点。现有的代码智能体通常依赖复杂的控制基架（Harness）来进行工具集成与执行反馈，然而这些环境在强化学习训练过程中往往面临崩溃率高、奖励攻击（Reward Hacking）严重以及训练与推理不一致等痛点。

为了解决这一行业难题，本文介绍了全新的 LEGO-RL 框架。该框架通过引入进程内大模型代理（In-process LLM Proxying）、可扩展的沙箱编排以及集成式诊断插件，成功实现了在不修改现有基架内部控制流的前提下进行策略梯度优化。实验表明，LEGO-RL 在多个主流基准测试（如 SWE-bench Verified）中显著提升了模型性能，同时保持了极高的训练稳定性与策略保真度，为代码智能体的规模化强化学习开辟了一条新路径。

LEGO-RL: Harness-Native Reinforcement Learning for Coding Agents

arXiv: 2608.17393
Date: August 18, 2026
Subject: Artificial Intelligence (cs.AI)

arXiv: 2608.17393
Date: August 18, 2026
Subject: Artificial Intelligence (cs.AI)

Summary

LEGO-RL is a novel framework designed to bridge the gap between native coding-agent harnesses and scalable reinforcement learning (RL). Current coding agents rely on complex harnesses for tool integration and execution feedback, but these environments often suffer from crashes, reward hacking, and train-inference discrepancies. LEGO-RL enables policy-gradient optimization without requiring modifications to the internal control flow of existing harnesses. By utilizing in-process LLM proxying, scalable sandbox orchestration, and integrated diagnostic plugins, the framework significantly improves performance across multiple coding benchmarks while maintaining high training stability.

LEGO-RL 是一个旨在弥合原生代码智能体基架与可扩展强化学习（RL）之间鸿沟的新型框架。当前的代码智能体依赖复杂的基架进行工具集成和执行反馈，但这些环境经常面临崩溃、奖励攻击以及训练-推理不一致等问题。LEGO-RL 能够在无需修改现有基架内部控制流的情况下，实现策略梯度优化。通过利用进程内大模型代理、可扩展的沙箱编排以及集成的诊断插件，该框架在保持高训练稳定性的同时，显著提升了多个代码基准测试的性能。

Key Pillars of LEGO-RL

  1. Faithful Optimization: Uses in-process LLM proxying to capture raw generation streams. This allows for token-level alignment and robust trainer-side log-probability recomputation, ensuring consistency even when the harness performs compaction or re-serialization.
  2. Reliable Execution: Features scalable sandbox orchestration that incorporates image caching and stage-wise defenses, effectively mitigating reward hacking during the training process.
  3. Observable Training: Includes an integrated plugin for automated validation and monitoring, complemented by a "Live UI" that provides developers with granular diagnostics of agent trajectories.

LEGO-RL 的三大核心支柱

  1. 高保真优化（Faithful Optimization）： 使用进程内大模型代理来捕获原始生成流。这实现了词元级别的对齐和稳健的训练端对数概率重新计算，确保即使在基架执行压缩或重新序列化时也能保持一致性。
  2. 可靠执行（Reliable Execution）： 具备可扩展的沙箱编排功能，结合了镜像缓存和分阶段防御，有效缓解了训练过程中的奖励攻击。
  3. 可观测训练（Observable Training）： 包含用于自动化验证和监控的集成插件，并辅以“实时用户界面（Live UI）”，为开发者提供智能体轨迹的精细化诊断。

Performance Highlights

LEGO-RL was evaluated by training the sparse MoE model Qwen3.5-35B-A3B using GSPO (Generative Sequence Policy Optimization). The framework demonstrated significant performance gains on the SWE-bench Verified benchmark:

  • OpenHands SDK: Improved from 64.0% to 70.4%.
  • Claude Code: Improved from 62.4% to 68.2%.
  • OpenCode: Improved from 57.2% to 66.6%.

Notably, the framework achieved a rollout-training probability correlation of over 0.99, indicating high fidelity between the training process and actual agent behavior.

性能亮点

LEGO-RL 通过使用 GSPO（生成序列策略优化，Generative Sequence Policy Optimization）训练稀疏 MoE 模型 Qwen3.5-35B-A3B 进行了评估。该框架在 SWE-bench Verified 基准测试中展现出显著的性能提升：

  • OpenHands SDK： 从 64.0% 提升至 70.4%。
  • Claude Code： 从 62.4% 提升至 68.2%。
  • OpenCode： 从 57.2% 提升至 66.6%。

值得注意的是，该框架实现了超过 0.99 的 rollout-training 概率相关性，表明训练过程与实际智能体行为之间具有高度的保真度。

Access & Resources

访问与资源

Note: This content includes references to the original arXiv submission. Please refer to the official arXiv page for the most up-to-date information and bibliographic tools.

license icon

注：本文内容包含对原始 arXiv 提交的引用。有关最新信息和文献计量工具，请参考官方 arXiv 页面。

license icon