当护栏看似有效时：LLM智能体商业评估中的构念效度失效

文章背景与核心概要

随着大语言模型（LLM）智能体在交互式市场模拟中的广泛应用，研究人员经常使用价格、利润和福利增长等严谨的经济学指标来评估AI策略，但这往往会制造一种评估严密的错觉，而实际上并未真正测试出所声称的底层行为。本文深入探讨了可配置酒店交易的多轮买卖双方测试平台中存在的构念效度（construct validity）失效问题。作者指出，市场护栏（marketplace guardrails）的朴素实现可能会由于混淆实验变量（例如不一致的报价架构和选择程序）而产生看似巨大的福利提升，但这并非源于真实的政策有效性。

为了解决这一问题，该研究引入了一项“构念效度契约”（construct-validity contract），要求在提出任何有效的政策主张之前，必须对激励有效性、协议隔离、随机稳定性和福利核算进行严格的检查。这项研究并非证明护栏无效，而是揭示了一个关键事实：在模拟智能体和测试协议通过严格的构念效度检查之前，其表面上的经济价值将完全无法得到证实。

📌 执行摘要

Evaluating AI policies using interactive market simulations with Language Model (LLM) agents often creates the illusion of rigorous economic metrics—such as prices, prices, and welfare gains—without actually testing the underlying behavior claimed.

使用大语言模型（LLM）智能体通过交互式市场模拟来评估AI政策，往往会制造一种严谨经济指标（如价格、利润和福利增长）的错觉，而实际上并没有真正测试所声称的底层行为。

This paper investigates construct validity failures in multi-turn buyer-seller testbeds for configurable hotel transactions. The authors demonstrate that naive implementations of marketplace guardrails can produce seemingly massive welfare improvements simply due to confounding experimental variables (like inconsistent offer schemas and choice procedures) rather than true policy effectiveness.

本文调查了可配置酒店交易的多轮买卖双方测试平台中的构念效度失效问题。作者证明，市场护栏的朴素实现可能会产生看似巨大的福利改善，但这仅仅是由于混淆了实验变量（如不一致的报价架构和选择程序），而不是真正的政策有效性。

To address this, the study introduces a construct-validity contract requiring rigorous checks on incentive validity, protocol isolation, stochastic stability, and welfare accounting before any valid policy claims can be made.

为了解决这一问题，该研究引入了一项构念效度契约（construct-validity contract），要求在做出任何有效的政策主张之前，必须对激励有效性、协议隔离、随机稳定性和福利核算进行严格的检查。

🔍 Key Findings & Audit Results

The Illusion of Policy Gains:
An initial implementation of two marketplace guardrails reported substantial welfare gains (+87.4, +35.0, and +28.8) across a Qwen2.5 model scale ladder (1.5B to 14B).
The Confounding Variable:
The initial test gave guarded and unguarded agents fundamentally different offer schemas and choice procedures. When these elements were held strictly fixed, the paired contrasts shifted dramatically (+7.2, -13.9, and +23.8).
Generation Variance:
The four largest 14B single-generation effects averaged +229. However, after running three generations per profile-condition, the average dropped to +37.6 (95% bootstrap interval [-34.2, 109.3]), with generation residuals accounting for 49.9% of the variation.
Non-Monotone Incentive Checks:
Increasing profit pressure on sellers produced less profit than the default seller prompt, exposing flaws in seller-incentive stability.
Positive Controls:
A profit-maximizing seller naturally attains the first-best welfare outcome. Consequently, guardrails primarily redistribute and can even reduce overall welfare, creating positive welfare effects only when sellers are explicitly programmed to force inefficient bundles.

🔍 核心发现与审计结果

政策收益的幻觉：
在Qwen2.5模型规模阶梯（从1.5B到14B）中，两个市场护栏的初始实现报告了显着的福利收益（分别为 +87.4、+35.0 和 +28.8）。

混淆变量：
初始测试为受保护和未受保护的智能体赋予了根本不同的报价架构和选择程序。当这些要素被严格保持固定时，成对的对比结果发生了戏剧性的变化（分别为 +7.2、-13.9 和 +23.8）。

生成方差：
四个最大的14B单代（single-generation）效应平均为 +229。然而，在每个配置文件-条件组合运行三代之后，平均值下降到 +37.6（95%自助法区间为 [-34.2, 109.3]），其中生成残差占变异的 49.9%。

非单调激励检查：
增加对卖方的利润压力反而产生了比默认卖方提示词更少的利润，暴露了卖方激励稳定性的缺陷。

阳性对照（Positive Controls）：
追求利润最大化的卖方自然能达到最优（first-best）的福利结果。因此，护栏主要起到重新分配的作用，甚至可能降低整体福利，只有在明确编程要求卖方强制推送低效捆绑包时，才会产生积极的福利效果。

🛠️ The Proposed Construct-Validity Contract

The authors argue that simulated agent environments must pass four foundational validation gates before returning substantive policy claims (classifying them as INVALID or INCONCLUSIVE otherwise):

Incentive Validity: Ensuring economic actors behave predictably under varied incentive structures.
Protocol Isolation: Guaranteeing that control and test groups share identical offer schemas and choice protocols.
Stochastic Stability: Accounting for generational variance and prompt stochasticity rather than relying on single-run artifacts.
Welfare Accounting: Rigorously tracking true economic efficiency over mere output formatting.

🛠️ 提出的构念效度契约

作者认为，模拟智能体环境在返回实质性政策主张之前，必须通过四个基础的验证关卡（否则将被归类为无效 (INVALID) 或 不确定 (INCONCLUSIVE)）：

激励有效性（Incentive Validity）： 确保经济主体在不同的激励结构下表现出可预测的行为。

协议隔离（Protocol Isolation）： 保证对照组和测试组共享相同的报价架构和选择协议。

随机稳定性（Stochastic Stability）： 考虑代际方差和提示词随机性，而不是依赖单次运行的产物。

福利核算（Welfare Accounting）： 严格追踪真正的经济效率，而非单纯的输出格式化。

Conclusion of the Case Study

Original Estimate: Marked as INVALID due to failures in protocol isolation.
Controlled Study: Remained INCONCLUSIVE due to remaining vulnerabilities in incentive validity and stochastic stability.

Ultimately, the study does not prove that guardrails are ineffective, but rather that their apparent economic value remains completely unidentified until simulated agents and testing protocols pass rigorous construct validity checks.

案例研究结论

原始估计： 由于协议隔离失败，被标记为无效 (INVALID)。

受控研究： 由于在激励有效性和随机稳定性方面仍存在漏洞，结果保持不确定 (INCONCLUSIVE)。

归根结底，这项研究并没有证明护栏是无效的，而是表明：在模拟智能体和测试协议通过严格的构念效度检查之前，其表面的经济价值将完全无法确认。