文章背景与核心概要
随着自主智能体（Agentic）AI系统迅速占领现有的软件工程基准测试，由于缺乏对任务难度的控制和表征，解读它们的真实性能仍然面临巨大挑战。《是什么让软件问题解决任务对智能体来说如此困难？》（What Makes Software Issue Resolution Tasks Difficult for Agents?）一文深入研究了决定某个任务对智能体而言比另一个任务更难的结构与语言属性。
通过使用迄今为止最大的开源编码智能体轨迹数据集 CoderForge-Preview 进行大规模实证研究，作者评估了是否可以利用任务补丁（patch）、代码库（repository）以及提示词（prompt）中的静态特征来预测任务的结果。研究表明，任务难度在很大程度上是可预测的（AUC = 0.863），这主要受补丁碎片化（patch fragmentation）和代码库规模（repository scale）的驱动，而提示词的语言特征则在中等难度任务中起着关键作用。这项研究为事前（pre-hoc）难度评估以及构建难度可控的基准测试奠定了基础。
What Makes Software Issue Resolution Tasks Difficult for Agents?
Authors: Ebtesam Al-Haque, Brittany Johnson
Published: August 18, 2026 (To appear in ESEM 2026)
Subjects: Software Engineering (cs.SE); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG)
arXiv: 2608.18280 | DOI: 10.48550/arXiv.2608.18280
(View License)
📌 Summary
As agentic AI systems rapidly saturate existing software engineering benchmarks, interpreting their true performance remains challenging due to a lack of control and characterization regarding task difficulty. What Makes Software Issue Resolution Tasks Difficult for Agents? investigates the structural and linguistic properties that determine why one task is harder for an agent than another.
Through a large-scale empirical study using CoderForge-Preview—the largest open dataset of coding agent trajectories—the authors evaluated whether task outcomes can be predicted using static features from task patches, repositories, and prompts. The study reveals that task difficulty is largely predictable (AUC = 0.863), driven primarily by patch fragmentation and repository scale, with prompt linguistic features playing a key role in mid-band tasks. This research establishes a foundation for pre-hoc difficulty estimation and difficulty-controlled benchmark construction.
随着自主智能体AI系统迅速占领现有的软件工程基准测试，由于缺乏对任务难度的控制和表征，解读它们的真实性能仍然是一个挑战。《是什么让软件问题解决任务对智能体来说如此困难？》一文研究了决定为什么某个任务对智能体而言比另一个任务更难的结构和语言属性。
通过使用迄今为止最大的开源编码智能体轨迹数据集 CoderForge-Preview 进行大规模实证研究，作者评估了是否可以使用来自任务补丁、代码库和提示词的静态特征来预测任务结果。研究表明，任务难度在很大程度上是可预测的（AUC = 0.863），主要由补丁碎片化和代码库规模驱动，而提示词语言特征在中等难度任务中发挥着关键作用。这项研究为事前难度估计和构建难度可控的基准测试奠定了基础。
📖 Abstract
- Background: Advances in agentic systems are simultaneously, and rapidly, saturating benchmarks. Despite this often-discussed phenomenon, benchmark scores remain difficult to interpret due to the lack of control and characterization of task difficulty. More specifically, we currently have little understanding of what makes one task harder than another, and to what extent task difficulty is predictable from static task properties.
- Aims: We propose a measurement framework to investigate and systematically quantify what structural properties of software tasks correspond to agent success rates for issue resolution tasks.
- Method: We conducted a large scale empirical study on CoderForge-Preview, the largest open dataset of coding agent trajectories to date, by extracting features across task patch, repository and prompt. We evaluated the predictive power of each feature against task outcomes using ensemble methods, SHAP attribution, and effect size analysis.
- Results: We found that task difficulty is substantially predictable from static features (AUC = 0.863) and is largely driven by patch fragmentation and repository scale. Prompt linguistic features become visible among top contributors for tasks in the mid-band, revealing a layered structure of difficulty.
- Conclusion: The difficulty of an issue resolution task is encoded in its structure. This enables static, pre-hoc difficulty estimation and lays the groundwork for difficulty-controlled benchmark construction for evaluation of agents.
- 背景： 智能体系统的进步正在同步且迅速地使基准测试饱和。尽管这一现象经常被讨论，但由于缺乏对任务难度的控制和表征，基准测试得分仍然难以解读。更具体地说，我们目前对“是什么使一个任务比另一个任务更难”以及“从静态任务属性中预测任务难度的程度如何”了解甚少。
- 目标： 我们提出了一个测量框架，旨在研究并系统量化软件任务的哪些结构属性对应于问题解决任务中智能体的成功率。
- 方法： 我们对迄今为止最大的编码智能体轨迹开源数据集 CoderForge-Preview 进行了大规模实证研究，通过提取任务补丁、代码库和提示词的特征，利用集成方法、SHAP 归因和效应量分析评估了每个特征对任务结果的预测能力。
- 结果： 我们发现，任务难度在很大程度上可以根据静态特征进行预测（AUC = 0.863），并且主要受补丁碎片化和代码库规模的驱动。在中等难度的任务中，提示词语言特征成为主要的贡献因素之一，揭示了难度的分层结构。
- 结论： 问题解决任务的难度编码在其结构中。这使得静态的事前难度估计成为可能，并为构建用于评估智能体的难度可控基准测试奠定了基础。