跳转至

文章背景与核心概要

在开源软件安全任务中，AI智能体的表现日益出色，然而分析二进制软件（如恶意软件、固件和专有应用程序）仍然是一个关键的瓶颈。二进制分析要求通过逆向工程（RE）来恢复程序语义，然后才能进行有意义的安全评估。为了在没有数据污染或人工简化的前提下严格测试AI在该领域的真实能力，研究人员推出了 SRE-Bench——首个现实且无污染的逆向工程基准。

通过对五款前沿大语言模型（LLM）的评估发现，目前的AI智能体在二进制逆向工程方面仍显力不从心，这凸显了智能体网络安全未来亟待攻克的重要前沿方向。该基准由逆向工程专家耗费超5000小时完全从零构建，包含19个私有、真实规模的程序及数百个二进制实例，为衡量未来智能体网络安全的发展提供了一个严苛的测试平台。

The Next Challenge for Agentic Cybersecurity: A Realistic, Contamination-Free Reverse Engineering Benchmark

license icon

  • arXiv ID: arXiv:2608.11469 [cs.CR]
  • Submitted: August 11, 2026
  • Subjects: Cryptography and Security (cs.CR); Artificial Intelligence (cs.AI); Software Engineering (cs.SE)
  • Authors: Jeremy Spence, Nicholas Assaderaghi, Jinhao Zhu, Nikil Ravi, Raluca Ada Popa, Guannan Wei, Yangruibo Ding, Zhuo Zhang

📌 Executive Summary

AI 智能体在处理有源码可查的软件安全任务时表现优异，然而，许多对网络安全至关重要的软件（如恶意软件、固件和专有应用程序）往往仅提供二进制文件。二进制分析要求进行逆向工程（RE）：即在进行有意义的分析之前恢复程序语义。

为了在没有数据污染或人工简化的前提下严格测试 AI 在该领域的各项能力，研究人员引入了 SRE-Bench——首个现实且无污染的逆向工程基准。对五款前沿大语言模型（LLM）的评估表明，当前的 AI 智能体在二进制逆向工程领域仍未得到很好解决，这凸显了智能体网络安全的重要发展前沿。

While AI agents are excelling in software security tasks when source code is available, analyzing binary software (such as malware, firmware, and proprietary applications) remains a critical bottleneck. Binary analysis requires Reverse Engineering (RE) to recover program semantics before meaningful evaluation can occur.

To rigorously test AI capabilities in this domain without data contamination or artificial simplifications, researchers introduce SRE-Bench—the first realistic, contamination-free reverse engineering benchmark. Evaluations across five frontier Large Language Models (LLMs) reveal that binary reverse engineering remains largely unsolved by current AI agents, highlighting a major frontier for agentic cybersecurity.

👥 Authors

  • Jeremy Spence
  • Nicholas Assaderaghi
  • Jinhao Zhu
  • Nikil Ravi
  • Raluca Ada Popa
  • Guannan Wei
  • Yangruibo Ding
  • Zhuo Zhang

📖 Abstract

当提供分析源码时，AI 智能体在网络安全能力上的提升速度极快，然而，大部分对网络安全至关重要的软件（包括恶意软件、固件和专有应用程序）却只能以二进制形式获得。分析此类软件需要进行逆向工程（RE）：即在开展有意义的分析之前恢复程序的语义。

然而，评估基于智能体的逆向工程面临着一个根本性挑战：基准实例必须是 LLM 训练数据中未曾作为源码出现过的“未见”内容，以防止模型通过死记硬背或识别走捷径而不是真正地去分析；同时，它还要匹配真实软件的规模与反分析保护措施。

遗憾的是，现有的基准无法同时满足这些要求。为此，作者推出了 SRE-Bench，这是首个现实且无污染的 RE 基准。SRE-Bench 由 RE 专家耗费超过 5000 小时从零完全构建，包含： * 19 个私有的、现实世界规模的程序，平均代码行数达 16,900 行。 * 44 种内部反分析基元。 * 262 个二进制实例。 * 1,572 个确定性评分任务。

AI agents are rapidly improving in cybersecurity capabilities when the source code is available for analysis, yet much of the software most consequential to cybersecurity, including malware, firmware, and proprietary applications, is available only as binaries. Analyzing such software requires reverse engineering (RE): recovering program semantics before the analysis can be meaningfully performed.

However, evaluating agentic RE poses a fundamental challenge: benchmark instances must be unseen as source code in the LLMs' training data to prevent models from taking shortcuts by recognizing them rather than really analyzing them, while also matching the scale and anti-analysis protections of real software.

Unfortunately, existing benchmarks do not jointly satisfy these requirements. To this end, the authors introduce SRE-Bench, the first realistic, contamination-free RE benchmark. Built entirely from scratch by RE experts with over 5,000 hours of effort, SRE-Bench comprises: * 19 private, real-world-scale programs averaging 16,900 lines of code. * 44 in-house anti-analysis primitives. * 262 binary instances. * 1,572 deterministically graded tasks.

🧪 Benchmark Results & Key Findings

评估涵盖了五款前沿 LLM：GPT-5.6-sol、Claude-Opus-5、GPT-5.5、Grok-4.5 和 GLM-5.2。

  • 成功率有限： 逆向工程在很大程度上仍未被攻克。最强的模型 GPT-5.6-sol 在每个实例上的平均得分率为 61.4%，且仅能完全解决 31.5% 的实例。
  • 行为差异： 智能体的表现与人类工程师不同；值得注意的是，AI 智能体对编译器优化和静态链接相对不敏感。
  • 消融实验洞察： 控制变量的消融实验证实，污染控制和现实规模对于准确测量能力缺一不可。

结论： 强大的源码安全能力尚不能直接迁移到二进制分析中。SRE-Bench 作为一个严苛的测试平台，可用于衡量智能体网络安全未来的进展。

Evaluations were conducted across five frontier LLMs: GPT-5.6-sol, Claude-Opus-5, GPT-5.5, Grok-4.5, and GLM-5.2.

  • Limited Success: RE remains largely unsolved. The strongest model, GPT-5.6-sol, scores 61.4% per instance and fully solves only 31.5% of the instances.
  • Behavioral Differences: Agents behave differently from human engineers; notably, AI agents are relatively insensitive to compiler optimization and static linking.
  • Ablation Insights: Controlled ablations confirm that both contamination control and realistic scale are essential for accurate capability measurement.

Conclusion: Strong source-code security capabilities do not yet transfer to binary analysis. SRE-Bench serves as a rigorous testbed to measure future progress in agentic cybersecurity.