跳转至

多套具现强化学习学到了什么？编码智能体中的归因与可迁移性

文章背景与核心概要

本文深入探讨了多套具现强化学习（Multi-Harness Reinforcement Learning, RL）在仓库级编码智能体中的工作机制。具体而言，作者隔离了跨多个不同执行环境（Aider、OpenHands、Qwen Code 和 SWE-agent）的群组相对策略优化（GRPO）技术，以评估信用分配（credit assignment）、配置适配以及真正的能力可迁移性。

研究发现，评估环境（Evaluation Harness）的选择是决定性能的主导因素，其对平均解决率的影响远超训练配方本身；而不同的分组规则（组内对比与跨组混合对比）对未见过的环境性能影响微乎其微。这表明，跨环境的信用分配主要带来的是针对特定环境的配置适配，而非本质上更具可迁移性的能力。该研究为未来的多套具现强化学习研究提供了重要的基准和方法论建议。

📋 执行摘要 (Executive Summary)

本文研究了多套具现强化学习（Multi-Harness RL）在代码库级别编码智能体中的运作方式。具体而言，作者隔离了多个不同执行环境（Aider、OpenHands、Qwen Code 和 SWE-agent）中的群组相对策略优化（GRPO）技术，以评估信用分配、配置适配以及真正的能力可迁移性。

核心发现：

  • 评估环境起主导作用： 在 24,000 次封闭评估中，评估环境的选择使平均解决率从 2.14% 跃升至 9.27%（相差 4.3 倍），其影响显著超过了训练配方本身的影响（1.16 倍）。
  • 分组规则影响甚微： 比较组内（每个任务-环境对一个组）与跨组（在任务内合并环境）的分组策略，发现它们在保留环境（held-out harnesses）上的表现差异微乎其微（跨组合并策略仅高出 +0.25 个百分点，且 95% 置信区间重叠）。
  • 配置适配与核心可迁移性的对比： 跨环境信用分配带来的是对特定环境的配置适配，并没有比环境内信用分配产生根本上更具可迁移性的能力。
  • 建议： 未来的多套具现强化学习研究必须明确声明分组边界，并针对未见过的评估环境测试其性能。

This paper investigates how Multi-Harness Reinforcement Learning (RL) functions in repository-level coding agents. Specifically, the authors isolate group-relative policy optimization (GRPO) techniques across multiple distinct execution harnesses (Aider, OpenHands, Qwen Code, and SWE-agent) to evaluate credit assignment, configuration adaptation, and true capability portability.

Key Findings:

  • The Evaluation Harness is Dominant: Across 24,000 sealed evaluations, the choice of evaluation harness moved the mean solve rate from 2.14% to 9.27% (a 4.3× factor), significantly overshadowing the training recipe's impact (1.16×).
  • Grouping Rules Matter Little: Comparing Within (one group per task-harness pair) vs. Cross (harnesses pooled within a task) grouping strategies revealed negligible differences on held-out harnesses (+0.25 pp for Cross, with overlapping 95% confidence intervals).
  • Configuration Adaptation vs. Core Portability: Cross-harness credit assignment yields configuration adaptation to specific environments rather than fundamentally more portable capabilities than within-harness credit.
  • Recommendation: Future multi-harness RL studies must explicitly state grouping boundaries and test performance against unseen evaluation harnesses.

📄 摘要 (Abstract)

智能体强化学习（RL）越来越多地通过完整的执行环境（harnesses）运行，而多套具现配方（multi-harness recipe）混合了两种选择：将策略暴露给多个环境，并在一个相对优势群组内比较它们的奖励。我们在仓库级编码中隔离了第二种选择。

从一个 Qwen3-8B 监督热启动（supervised warm start）开始，我们在 Aider、OpenHands、Qwen Code 和 SWE-agent 上重放了相同的冻结任务-环境记录，在相同的更新次数下，遵循群组相对策略优化（GRPO）的两条规则： 1. 组内（Within）（每个任务-环境对一个组） 2. 跨组（Cross）（在任务内合并环境）

每个检查点都通过封闭的 SWE-bench Verified 预言机（oracle）在四个源环境和一个训练中保留的极简环境上进行评分。

评估环境是起主导作用的变量：在 24,000 次封闭评估中，它将平均解决率从 \(2.14\%\) 提升至 \(9.27\%\)（相差 \(4.3\) 倍），而训练配方仅将其改变 \(1.16\) 倍。分组规则则不然。在保留环境上，在每项任务尝试八次的条件下，Cross 减去 Within 的差值为 \(+0.25\text{ pp}\)（\(95\%\) 置信区间 \([-0.48, +1.02]\)），而在跨越三个训练种子（其各自的估计值符号会发生变化）进行合并后，该值为 \(+0.16\text{ }[-0.41, +0.72]\)。每条规则自身的种子变动范围（\(0.42\) 到 \(0.45\text{ pp}\)）甚至超过了它们之间的差异。

两条规则都在相同的源环境上取得了最大的收益。合并后的优势支撑起了该环境：一个域外（out-of-fold）分类器能够从 Cross 的优势中恢复出生成环境，其准确率比打乱标签的基线高出 \(+4.48\text{ pp}\)，而对于 Within 则完全无法恢复；并且这两条规则在每个环境内部仍然达到了相同的保留分数和动作分布。在策略上重新收集一半的训练数据并不会改变这一点。

跨环境信用分配产生了配置适配，但并没有带来比环境内信用分配更多可迁移的能力。多套具现 RL 报告应当说明分组边界，并在未见过的环境进行测试。

Agent reinforcement learning (RL) increasingly runs through full execution harnesses, and a multi-harness recipe mixes two choices: exposing the policy to several harnesses, and comparing their rewards inside one relative-advantage group. We isolate the second choice in repository-level coding.

From one Qwen3-8B supervised warm start we replay the same frozen task-harness records from Aider, OpenHands, Qwen Code, and SWE-agent, with the same number of updates, under two rules for group-relative policy optimization (GRPO): 1. Within (one group per task-harness pair) 2. Cross (harnesses pooled within a task)

Every checkpoint is scored with a sealed SWE-bench Verified oracle on four source harnesses and a minimal harness held out of training.

The evaluation harness is the dominant variable: across 24,000 sealed evaluations it moves the mean solve rate from \(2.14\%\) to \(9.27\%\), a factor of \(4.3\), where the training recipe moves it by \(1.16\). The grouping rule is not. On the held-out harness, Cross minus Within is \(+0.25\text{ pp}\) (\(95\%\) confidence interval \([-0.48, +1.02]\)) at eight attempts per task, and \(+0.16\text{ }[-0.41, +0.72]\) pooled over three training seeds whose individual estimates change sign. Each rule's own seed range, \(0.42\) to \(0.45\text{ pp}\), exceeds the difference between them.

Both rules place their largest gains on the same source harness. The pooled advantage carries the harness: an out-of-fold classifier recovers the generating harness from Cross's advantage \(+4.48\text{ pp}\) above the shuffled-label baseline and from Within's not at all, and the two rules still reach the same held-out score and action distribution inside each harness. Re-collecting half the training data on-policy does not change this.

Cross-harness credit yields configuration adaptation and no more portable capability than within-harness credit. Multi-harness RL reports should state the grouping boundary and test under an unseen harness.