文章背景与核心概要
随着供应商逐渐淘汰旧模型，依赖商业大语言模型（LLM）API的软件系统经常面临向后续版本迁移的挑战。传统上，这些迁移决策主要由聚合基准评分（即压缩了复杂、异构的项目级行为的单一净分值）来驱动。
本文深入探讨了这些聚合评分所掩盖的真相。作者通过使用 900 个公共基准测试项目，对 GPT-5.4 到 GPT-5.6 Sol 产品序列中的三次成对升级进行了评估，揭示了聚合指标掩盖了大量双向的项目级变化。即使是表现出净提升的模型也隐藏着显著的性能退化，反之亦然，这表明仅仅依赖聚合评分会给软件可靠性引入隐性风险。
聚合评分的盲区：衡量商业大模型API迁移中的项目级性能退化
作者： Xiaonan Xu, Wenjing Wu
发布时间： 2026年8月18日
主要学科： 软件工程 (cs.SE)
其他学科： 人工智能 (cs.AI)、计算与语言 (cs.CL)
arXiv ID： 2608.17719
DOI： 10.48550/arXiv.2608.17719
执行摘要 (Executive Summary)
Software systems relying on commercial Large Language Model (LLM) APIs frequently face the challenge of migrating to successor versions when older models are deprecated by vendors. Traditionally, these migration decisions are driven by aggregate benchmark scores—single net figures that compress complex, heterogeneous item-level behaviors.
依赖商业大语言模型（LLM）API的软件系统在供应商淘汰旧模型时，经常面临向后续版本迁移的挑战。传统上，这些迁移决策由聚合基准评分驱动——即那些将复杂、异构的项目级行为压缩成单一净值的数字。
This paper investigates what those aggregate scores conceal. By evaluating three pairwise upgrades across the GPT-5.4 to GPT-5.6 Sol product sequence using 900 public benchmark items, the authors reveal that aggregate metrics mask substantial bidirectional item-level changes. Even models showing net improvements harbor significant regressions, and vice versa, demonstrating that relying solely on aggregate scores introduces hidden risks to software reliability.
本文调查了这些聚合评分所掩盖的内容。通过使用 900 个公共基准测试项目评估 GPT-5.4 到 GPT-5.6 Sol 产品序列中的三次成对升级，作者揭示了聚合指标掩盖了大量的双向项目级变化。即使是显示出净提升的模型也包含显著的退化，反之亦然，这表明仅依赖聚合评分会给软件可靠性带来隐藏的风险。
核心发现与方法论 (Key Findings & Methodology)
- Evaluation Scope: Tested across nine migration-benchmark cells covering graduate-level knowledge, olympiad mathematics, and instruction following.
- Testing Rigor: Queried 900 public benchmark items 50 times per item per model.
- Classification Criteria: Items were classified under false-discovery-rate control and a practical-significance threshold into four categories:
- Reliably improved
- Reliably regressed
- Practically equivalent
- Inconclusive
- 评估范围： 测试了涵盖研究生水平知识、奥林匹克数学和指令遵循的九个迁移基准单元。
- 测试严格性： 每个模型对 900 个公共基准项目中的每一个查询了 50 次。
- 分类标准： 在错误发现率控制和实际显著性阈值下，项目被分为四类：
- 可靠提升
- 可靠退化
- 实际等效
- 结论不确定
- Coexistence of Improvements and Regressions: Across all tested migrations, reliable improvements and reliable regressions coexisted:
- Edges boasting aggregate gains of up to 7.3 percentage points still contained up to 8.3% reliably regressed items.
- Edges exhibiting aggregate losses still contained up to 10.7% reliably improved items.
- Scoring Sensitivity: On the instruction-following benchmark during the latest migration, the gap between strict and loose scoring widened by 3.9 percentage points (a 3.9-point regression under strict scoring shrank to a negligible 0.04 points under loose scoring).
- 提升与退化的共存： 在所有测试的迁移中，可靠的提升和可靠的退化同时存在：
- 宣称聚合增益高达 7.3 个百分点的迁移路径中，仍然包含高达 8.3% 的可靠退化项目。
- 表现出聚合损失的迁移路径中，仍然包含高达 10.7% 的可靠提升项目。
- 评分敏感性： 在最近一次迁移的指令遵循基准测试中，严格评分与宽松评分之间的差距扩大了 3.9 个百分点（在严格评分下表现出的 3.9 个百分点退化，在宽松评分下缩小至微不足道的 0.04 个百分点）。
结论与资源 (Conclusion & Resources)
Migration decisions based strictly on aggregate scores fail to capture hidden item-level regressions that can impact downstream software performance. To support reproducible research and safer migrations, the complete response-level archive and per-item scoring outputs have been publicly released.
严格基于聚合评分的迁移决策无法捕捉到可能影响下游软件性能的隐性项目级退化。为了支持可重复性研究和更安全的迁移，完整的响应级存档和逐项评分输出已向公众发布。
- Full-Text Access: View PDF | HTML Version
- Code & Data: Associated archives available via arXiv metadata and connected platforms (Hugging Face / CatalyzeX / Papers with Code).