跳转至

文章背景与核心概要

传统的单轮代码生成指标往往无法衡量实际部署的AI智能体的一个关键能力：即在获得具体反馈后修复错误输出的能力。本文引入了 A-CEGIS，这是一个轻量级的评估框架，它利用反例来指导自然语言到正则表达式（NL-to-regex）合成中的多轮精化过程。

该框架利用确定性预言机（deterministic oracle）在全匹配语义下验证提出的正则表达式，并提供紧凑的假阳性或假阴性见证反馈来引导后续轮次。实验结果表明，诊断性的反例反馈显著优于零样本生成和纯错误反馈，在四轮预算内成功解决了 90% 的任务，并通过针对性探测在隐藏评估集上实现了 100% 的成功率。

Counterexamples as Feedback for Agent Self-Counteraction

arXiv: 2609.02892 [cs.CL]
Authors: Sidhesh Badrinarayan, Adithya Parthasarathy
Submitted: July 1, 2026
Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)

📌 Executive Summary

传统单轮代码生成指标往往无法捕捉部署的AI智能体的一项关键能力：在提供具体反馈时修复有缺陷输出的能力。本文介绍了 A-CEGIS，这是一个轻量级的评估框架，利用反例来指导自然语言到正则表达式（NL-to-regex）合成中的多轮精化。

通过利用确定性预言机在全匹配语义下验证所提出的正则表达式，该框架提供紧凑的假阳性或假阴性见证反馈来引导后续轮次。实验结果表明，诊断性反例反馈极大地优于零样本生成和纯错误反馈，在四轮预算内成功解决了90%的任务，并在通过有针对性的探测时在隐藏评估集上取得了100%的成功率。

Traditional single-turn code-generation metrics often fail to capture a crucial capability of deployed AI agents: their ability to repair flawed outputs when provided with concrete feedback. This paper introduces A-CEGIS, a lightweight evaluation framework that leverages counterexamples to guide multi-turn refinement in natural-language-to-regex (NL-to-regex) synthesis.

By utilizing a deterministic oracle to verify proposed regular expressions under full-match semantics, the framework supplies compact false-positive or false-negative witness feedback to steer subsequent turns. Experimental results demonstrate that diagnostic counterexample feedback dramatically outperforms zero-shot generation and error-only feedback, successfully solving 90% of tasks within a four-turn budget and achieving 100% success on hidden evaluation sets with targeted probing.

📖 Abstract

单轮代码生成指标低估了部署智能体的一个核心属性：它们是否能够在收到具体反馈后修复错误的制品。本文提出了 A-CEGIS，这是一个轻量级框架，使用反例作为反馈，用于评估自然语言到正则表达式合成中的多轮精化。

智能体提出一个正则表达式，确定性预言机在全匹配语义下对其进行检查，紧凑的假阳性或假阴性见证指导下一轮操作。 * 在30个 NL-RX-Turk 任务中，诊断性反例反馈在四轮消融预算内解决了 90% 的任务，相比之下： * 零样本生成为 17%， * 通用自我修正为 27%， * 纯错误反馈为 23%。 * 在带有强化的完整诊断运行中，到最后一轮时 隐藏集上的所有任务都被解决。 * 该方法实现了平均 2.7轮 的成功所需时间，并在有针对性的探测后达到了 77% 的强大成功率。

这些结果表明，A-CEGIS有效地衡量了智能体在多轮之间改进的效率，同时在原始保留案例之外增加了一个实用的稳健性检查。

Single-turn code-generation metrics understate a central property of deployed agents: whether they can repair a wrong artifact after receiving concrete feedback. This paper presents A-CEGIS, a lightweight framework that uses counterexamples as feedback for evaluating multi-turn refinement in natural-language-to-regex synthesis.

An agent proposes a regex, a deterministic oracle checks it under full-match semantics, and compact false-positive or false-negative witnesses guide the next turn. * On 30 NL-RX-Turk tasks, diagnostic counterexample feedback solves 90% of tasks within a four-turn ablation budget, compared with: * 17% for zero-shot generation, * 27% for generic self-correction, and * 23% for error-only feedback. * In a full diagnostic run with hardening, all tasks are solved on the hidden set by the final turn. * The approach achieves a mean time-to-success of 2.7 turns and a robust success rate of 77% after targeted probing.

These results show that A-CEGIS effectively measures how efficiently an agent improves across turns while adding a practical robustness check beyond original held-out cases.