跳转至

使用大语言模型与编程语言语义预测程序退出代码

文章背景与核心概要

本研究探讨了大语言模型（LLM）究竟是真正理解了编程语言的语义，还是仅仅依赖于预训练期间学到的统计先验。为了测试这一点，作者引入了一项名为程序可执行性预测（Program Executability Prediction, PrEx）的新型评估任务。该任务旨在测试模型在给定语法和操作语义的情况下，能否判断程序在语义上是否有效或无效，并识别其违反了哪条形式化规则。通过在多种语义形式化方法、语义偏移以及不同程序复杂度下对开源代码大语言模型进行评估，研究表明，模型在很大程度上依赖于预训练先验，而不是应用显式的语义规则，尤其是在面对修改后的语义或随着程序复杂度增加时表现尤为明显。

This research paper investigates whether Large Language Models (LLMs) truly understand programming language semantics or if they simply rely on statistical priors learned during pre-training. To test this, the authors introduce Program Executability Prediction (PrEx), a novel evaluation task that tests whether models can determine if a program is semantically valid or invalid—and identify which formal rule it violates—given its syntax and operational semantics. Evaluating open-source coding LLMs across multiple semantic formalisms, shifts, and program complexities, the study reveals that models lean heavily on pre-training priors rather than applying explicit semantic rules, especially under modified semantics or as program complexity increases.

元数据与出版详情

  • arXiv ID: arXiv:2609.00579 [cs.PL]
  • 学科分类: 编程语言 (cs.PL); 人工智能 (cs.AI); 计算与语言 (cs.CL); 软件工程 (cs.SE)
  • ACM 类别: F.3.2; I.2.6
  • 备注说明: 已被 LMPL 2026 接受
  • 提交日期: 2026年9月1日
  • 作者:
  • Lara Marinov
  • Aditya Thimmaiah
  • Jayanth Srinivasa
  • Junyi Jessy Li
  • Milos Gligoric

Metadata & Publication Details

  • arXiv ID: arXiv:2609.00579 [cs.PL]
  • Subjects: Programming Languages (cs.PL); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Software Engineering (cs.SE)
  • ACM Classes: F.3.2; I.2.6
  • Comments: Accepted at LMPL 2026
  • Submission Date: September 1, 2026
  • Authors:
  • Lara Marinov
  • Aditya Thimmaiah
  • Jayanth Srinivasa
  • Junyi Jessy Li
  • Milos Gligoric

摘要

大语言模型（LLM）在代码生成和翻译等各种软件工程任务中表现出了熟练的能力。然而，其性能的一个关键局限性可能是它们对编程语言语义的理解（或缺乏理解）。即使给出了显式的语义，目前仍不清楚 LLM 是应用了这些规则，还是转而依赖预训练期间学到的先验。

我们通过一项名为程序可执行性预测（PrEx）的新任务来研究 LLM 究竟是依赖先验还是给定语义。该任务要求模型在给定程序语法和操作语义的情况下，预测程序在语义上是有效还是无效（如果无效，则指出它违反了哪条形式化规则）。由于 PrEx 同时需要有效和无效程序，我们构建了一个数据集，其中包含从有效程序中系统生成的不合法转换。

我们在两种语义形式化方法和两种语义偏移下，对人工编写（Human-Written）、LLM 翻译（LLM-Translated）以及模糊测试生成（Fuzzer-Generated）的程序切片进行了开源代码 LLM 的评估。我们的研究结果表明，LLM 倾向于依赖预训练先验，而不是系统性地应用给定的规则，在修改后的语义上表现尤其糟糕，并且随着程序复杂度的增加性能进一步下降。

Abstract

Large language models (LLMs) have shown proficiency in various software engineering tasks, such as code generation and translation. However, a key limitation in their performance may be their (lack of) understanding of programming-language semantics. Even when explicit semantics are given, it remains unclear whether LLMs apply those rules or lean on priors learned during pre-training instead.

We study if LLMs lean on priors or given semantics with a novel task—Program Executability Prediction (PrEx)—that asks models to predict whether a program is semantically valid or invalid (and, if invalid, which formal rule it violates) given the program's syntax and operational semantics. Because PrEx requires both valid and invalid programs, we build a dataset with systematically generated invalid transformations derived from valid programs.

We evaluate open-source coding LLMs under two semantic formalisms and two semantic shifts across Human-Written, LLM-Translated, and Fuzzer-Generated program splits. Our findings show that LLMs lean on pre-training priors rather than systematically applying the given rules, performing especially poorly on modified semantics and degrading further as program complexity increases.

资源与链接

(注：与本出版物关联的许可图标: license icon)

(Note: License icon associated with this publication: license icon)