文章背景与核心概要

A/B测试是科技行业中推出新功能和优化方案的行业标准，但它通常伴随着周期长、消耗工程资源以及占用真实用户流量等痛点。在这篇论文中，作者Stefan Hut和Lorenzo Masoero探讨了是否可以利用AI智能体来可靠地模拟A/B测试结果，从而在投入真实流量之前对候选方案进行初步筛选。

为了解决这一问题，作者将该场景形式化为模拟随机对照试验（Simulated Randomized Controlled Trial, S-RCT），并引入了一个不依赖特定智能体（agent-agnostic）的验证框架，该框架具备双层误差分解能力。通过在67个历史营销A/B测试上的验证，研究发现开箱即用的基础模型能够捕捉到正确的方向信号，但会高估效应幅度。然而，通过实施两阶段前测期校准协议（two-phase pre-period calibration protocol），均方预测误差减少了约77倍；而被试内设计（within-subject design）则将标准误差降低了约2.4倍。这项研究为利用AI智能体加速产品迭代和实验评估开辟了新的技术路径。

1. 标题与摘要 (Title & Abstract)

Can AI Agents Simulate A/B Test Outcomes? A Validation Framework for Agentic Experimentation

Can AI Agents Simulate A/B Test Outcomes? A Validation Framework for Agentic Experimentation

Summary

A/B testing is crucial for feature rollouts in tech, but it is slow, costly in engineering resources, and consumes real user traffic. This paper, by Stefan Hut and Lorenzo Masoero, explores whether AI agents can reliably simulate A/B test outcomes to vet candidate treatments beforehand.

The authors formalize this as a Simulated Randomized Controlled Trial (S-RCT) and introduce an agent-agnostic validation framework featuring a two-layer error decomposition. Tested on 67 historical marketing A/B tests, an out-of-the-box foundation model captured correct directional signals but overestimated effect magnitudes. However, implementing a two-phase pre-period calibration protocol reduced squared prediction error by ~77×, while a within-subject design reduced standard errors by ~2.4×.

Summary

A/B testing is crucial for feature rollouts in tech, but it is slow, costly in engineering resources, and consumes real user traffic. This paper, by Stefan Hut and Lorenzo Masoero, explores whether AI agents can reliably simulate A/B test outcomes to vet candidate treatments beforehand.

The authors formalize this as a Simulated Randomized Controlled Trial (S-RCT) and introduce an agent-agnostic validation framework featuring a two-layer error decomposition. Tested on 67 historical marketing A/B tests, an out-of-the-box foundation model captured correct directional signals but overestimated effect magnitudes. However, implementing a two-phase pre-period calibration protocol reduced squared prediction error by ~77×, while a within-subject design reduced standard errors by ~2.4×.

元数据与文档信息 (Metadata & Document Information)

Attribute	Details
arXiv Identifier	arXiv:2608.02345 [cs.CL]
Authors	Stefan Hut, Lorenzo Masoero
Subjects	Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Applications (stat.AP)
Submission History	• v1: Aug 3, 2026 • v2: Aug 13, 2026 (Last revised)
License	Creative Commons Attribution 4.0
Primary Links	• View PDF • HTML Version • DOI Reference

摘要 (Abstract)

在科技行业中，A/B测试仍然是推出新功能的行业标准。然而，每一项实验都会消耗真实的流量、工程开发成本以及数周的时钟时间。如果基于行为画像和干预措施的上下文描述，AI智能体能否准确地模拟实验结果，从而在投入真实流量之前对候选干预方案进行筛选？

我们将这个问题形式化为模拟随机对照试验（Simulated Randomized Controlled Trial, S-RCT），并推导出一个双层误差分解（two-layer error decomposition），该分解将智能体近似误差与子采样误差分离开来，从而能够针对性地改进每一部分。该框架与特定智能体无关：从微调的专业模型到通用基础模型，任何行为模型都可以充当模拟引擎。

在67个历史营销A/B测试中进行的验证表明，使用现成基础模型的基线S-RCT能够捕捉到方向信号（符号重合度为0.70），但系统性地高估了效应幅度： * 前测期校准： 两阶段前测期校准协议将平方预测误差（在去除不可约测量噪声后）减少了约 \(77\times\)。 * 被试内设计： 被试内设计（即让每个智能体都体验两组实验条件）将标准误差减少了约 \(2.4\times\)。

我们讨论了当前方法的局限性，并识别了实验者可以从智能体信号中受益的应用场景。

A/B testing remains the standard for rolling out new features in the technology industry. Each experiment, however, consumes real traffic, engineering effort, and weeks of wall-clock time. Can AI agents—conditioned on behavioral profiles and contextual descriptions of the intervention—simulate outcomes accurately enough to vet candidate treatments before committing live traffic?

We formalize this question as a Simulated Randomized Controlled Trial (S-RCT) and derive a two-layer error decomposition that separates agent approximation error from subsampling error, enabling targeted improvements to each. The framework is agent-agnostic: any behavioral model—from a fine-tuned specialist to a general-purpose foundation model—can serve as the simulation engine.

Validated on 67 historical marketing A/B tests, a baseline S-RCT using an off-the-shelf foundation model captures directional signal (sign overlap 0.70) but systematically overshoots effect magnitudes: * Pre-period calibration: A two-phase pre-period calibration protocol reduces the squared prediction error (after removing irreducible measurement noise) by \(\sim\!77\times\). * Within-subject design: A within-subject design—where each agent is exposed to both arms—reduces standard errors by \(\sim\!2.4\times\).

We discuss limitations of the current approach and identify applications where experimenters stand to benefit from agentic signals.

附加资源与工具 (Additional Resources & Tools)

• 代码与目录： 可通过 CatalyzeX Code Finder、Hugging Face 和 DagsHub 访问。
• 讨论与社区： alphaXiv、ScienceCast。
• 学术引用： Google Scholar、Semantic Scholar 以及 NASA ADS。

• Code & Catalogs: Accessible via CatalyzeX Code Finder, Hugging Face, and DagsHub.
• Discussion & Community: alphaXiv, ScienceCast.
• Academic References: Google Scholar, Semantic Scholar, and NASA ADS.