文章背景与核心概要

现代智能体系统越来越依赖模型声明的置信度来控制决策流程，并普遍假设高置信度与行动瞬间的正确性呈正相关。本文通过一种隐藏信息的国际象棋变体（其中“皇室”棋子可以被秘密且重复地重新定位），对这一假设进行了深入探究。

该研究通过在每一轮博弈中引导大语言模型（LLM）输出关于对手隐藏棋子的主观概率分布（独立于所选择的移动动作），并将其与游戏结束后可恢复的真实数据进行评分对比，从而评估了各大模型的表现。研究发现，大语言模型在面对隐藏信息时表现出严重的信念校准失误，尤其是在高置信度区间表现得淋漓尽致，这表明仅凭结果来评估智能体存在巨大的盲区。

Confident at the Moment of Action: Belief Miscalibration in LLM Play Under Hidden Information

Authors: Bhushan Kashinath Joshi
Submitted: August 25, 2026
Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG)
arXiv: 2608.24691 [cs.AI] | View PDF

Confident at the Moment of Action: Belief Miscalibration in LLM Play Under Hidden Information

Authors: Bhushan Kashinath Joshi
Submitted: August 25, 2026
Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG)
arXiv: 2608.24691 [cs.AI] | View PDF

📌 Summary

📌 Summary

Modern agentic systems increasingly rely on a model's stated confidence to gate decisions, assuming that high confidence correlates with correctness at the exact moment of action. This paper investigates that assumption using a hidden-information chess variant where a "royal" piece can be secretly and repeatedly relocated.

现代智能体系统越来越依赖模型声明的置信度来控制决策流程，并假设高置信度与行动瞬间的正确性呈正相关。本文通过一种隐藏信息的国际象棋变体（其中“皇室”棋子可以被秘密且重复地重新定位），对这一假设进行了探究。

The study evaluates Large Language Models (LLMs) by eliciting a stated probability distribution over the opponent's hidden piece on every turn (separate from the chosen move), which is then scored against ground-truth data recoverable post-game.

该研究通过在每一轮博弈中引导大语言模型（LLM）输出关于对手隐藏棋子的概率分布（独立于所选择的移动动作），并将其与游戏结束后可恢复的真实数据进行评分对比，从而对大语言模型进行了评估。

Key Findings:

Key Findings:

Severe Miscalibration at High Confidence: Across two independent evaluation batches, captures made at high stated confidence (\(\ge 0.5\)) regarding the hidden piece's location were correct in only 1 out of 62 cases.

高置信度下的严重校准失误： 在两个独立的评估批次中，针对隐藏棋子位置作出高置信度（\(\ge 0.5\)）声明所进行的捕获行动中，正确的案例在 62个中仅有 1 个。

Concentration of Deficit: This calibration failure is almost entirely isolated to high-confidence events (99.3% in the original batch, 98.7% in the replication).

缺陷的高度集中： 这种校准失效几乎完全孤立地发生在高效信度事件中（在原始批次中占 99.3%，在复制批次中占 98.7%）。

Impact of Deliberation Budgets: A same-model comparison at a fixed external leaderboard score demonstrates that altering just the deliberation budget shifts the calibration metric by nearly as much as a large cross-model gap.

审慎预算（Deliberation Budgets）的影响： 在固定的外部排行榜分数下进行同模型对比表明，仅改变审慎预算对校准指标的影响，就几乎可以媲美跨模型之间的大幅差距。

Dissociation from Conventional Metrics: Standard evaluation axes (legality, cost, latency, completion rate) can entirely decouple from belief quality. In some cases, the configuration winning on every conventional metric produced the worst belief quality tested.

与传统指标的脱节： 标准评估维度（合法性、成本、延迟、完成率）可能会与信念质量完全脱节。在某些情况下，在所有传统指标上都获胜的配置，却产生了测试中最差的信念质量。

The Invisible Flaw: Because a model exhibiting this pattern can still win the overall game, outcome-only evaluation completely fails to detect the underlying belief miscalibration.

不可见的缺陷： 由于表现出这种模式的模型仍然可以赢得整场比赛，仅看结果的评估方式完全无法检测到潜在的信念校准失误。