跳转至

文章背景与核心概要

大语言模型（LLM）在日常编程任务中表现优异，但在需要对程序语义进行深度推理的复杂任务中往往表现不佳。为了解决这类任务的训练数据生成难题，本文引入了一种基于语义不等价博弈（SInQ）的方法。在该框架中，生成器智能体从真实任务中创建语义上不同的程序变体，而评估器智能体则负责找出它们行为存在差异的输入。这些智能体通过半对抗的方式相互训练，从而增强对代码逻辑的理解。

在跨语言漏洞检测（仅在Python上训练，并在C/C++上进行测试）以及Python内置标识符交换基准测试等多个基准上的评估表明，该方法相比现代大语言模型取得了显著的性能提升。作者已公开复现代码及合成微调数据集，供学术界和工业界进一步研究使用。

Program Semantic Inequivalence Game with Large Language Models

Summary

Large Language Models (LLMs) excel at everyday coding tasks but often struggle with complex tasks requiring deep reasoning about program semantics. To address the challenge of generating training data for such tasks, this paper introduces a method based on a semantic inequivalence game (SInQ). In this setup, a generator agent creates semantically distinct program variants from real-world tasks, while an evaluator agent identifies inputs where their behaviors diverge. These agents train each other semi-adversarially to enhance code logic understanding. Evaluations on benchmarks—such as cross-language vulnerability detection (trained exclusively on Python, tested on C/C++) and the Python builtin identifier swap benchmark—demonstrate significant improvements over modern LLMs. The authors have released the replication code and synthetic fine-tuning data.

Paper Metadata

  • arXiv ID: arXiv:2505.03818 [cs.LG]
  • Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Programming Languages (cs.PL)
  • Authors:
  • Antonio Valerio Miceli-Barone
  • Vaishak Belle
  • Ali Payani
  • Submitted: 2 May 2025 (v1); Last revised: 12 Aug 2026 (v3)
  • DOI: 10.48550/arXiv.2505.03818

Abstract

大语言模型（LLM）能够在日常编码任务中取得出色的表现，但在需要对程序语义进行非平凡推理的复杂任务时可能会失效。寻找用于教导LLM解决这些任务的训练样本可能具有相当大的挑战性。

Large Language Models (LLMs) can achieve strong performance on everyday coding tasks, but they can fail on complex tasks that require non-trivial reasoning about program semantics. Finding training examples to teach LLMs to solve these tasks can be challenging.

在这项工作中，我们探索了一种基于语义不等价博弈（SInQ）来合成生成代码推理训练数据的方法：生成器智能体利用真实编程任务数据集创建语义不同的程序变体，而评估器智能体则必须找出它们行为表现不同的输入样本。这些智能体以半对抗的方式相互训练，从而提高了它们理解代码底层逻辑的能力。

In this work, we explore a method to synthetically generate code reasoning training data based on a semantic inequivalence game (SInQ): a generator agent creates program variants that are semantically distinct, derived from a dataset of real-world programming tasks, while an evaluator agent has to identify input examples for which they behave differently. The agents train each other semi-adversarially, improving their ability to understand the underlying logic of code.

我们在多个代码生成和理解基准测试上评估了我们的方法，包括跨语言漏洞检测（Lu et al., 2021）——尽管我们的方法仅在Python代码上进行训练，但它改善了C/C++代码中的漏洞检测——以及具有挑战性的Python内置标识符交换基准测试（Miceli Barone et al., 2023）。结果表明，尽管现代LLM在此基准测试上仍然举步维艰，但我们的方法带来了实质性的提升。

We evaluated our approach on multiple code generation and understanding benchmarks, including cross-language vulnerability detection (Lu et al., 2021), where our method improves vulnerability detection in C/C++ code despite being trained exclusively on Python code, and the challenging Python builtin identifier swap benchmark (Miceli Barone et al., 2023), showing that whereas modern LLMs still struggle with this benchmark, our approach yields substantial improvements.

我们公开了复现实验所需的代码，以及可用于微调LLM的生成合成数据。

We release the code needed to replicate the experiments, as well as the generated synthetic data, which can be used to fine-tune LLMs.

Access & Resources

Citation & References