跳转至

文章背景与核心概要

本文探讨了大语言模型（LLM）在推理过程中引入结构化机制（如规划、验证和修正循环）与Token消耗之间的权衡关系。虽然这些机制能让模型进行搜索和自我纠错，但它们也会占用宝贵的Token预算。研究人员通过在FinQA和TAT-QA财务推理任务上评估单一整体模型（Monolith LLM）与验证搜索架构（Verified Search Architecture），测试了从250到42,000个Token等14个预算档位。

研究发现了一个明确的临界交叉点：在1,000个Token以下，由于结构化开销挤占了响应空间，验证搜索的表现反而不如单一模型；而在1,000到1,500个Token之间出现了关键的交叉阈值（\(p \le 0.001\)）；当Token超过1,500时，验证搜索的性能开始超越单一模型，并在最高档位达到了约44%的准确率（单一模型约为40%）。该论文为理解大模型在推理预算受限时的系统架构设计提供了重要的量化依据。

Thinking Costs Tokens: When More Structure is Worth the Price

arXiv: 2608.27506 [cs.AI]
Submitted: 27 August 2026
Authors: Thomas Nolasque, John Grey, Calista Pham, Ankit Vani

arXiv: 2608.27506 [cs.AI]
Submitted: 27 August 2026
Authors: Thomas Nolasque, John Grey, Calista Pham, Ankit Vani

📌 Executive Summary

Adding inference structure (such as planning, verification, and revision loops) to large language models allows them to search and self-correct, but these processes consume valuable token budgets. This paper investigates whether a token-budget threshold exists below which structural overhead hurts performance and above which it helps.

Evaluating two systems (a monolithic LLM vs. a verified search architecture) on FinQA and TAT-QA financial reasoning tasks across 14 budget tiers (250 to 42,000 tokens), the authors discovered a clear crossover point: * Below 1,000 tokens: Structural overhead starves the model of response space, causing verified search to underperform. * Between 1,000 and 1,500 tokens: The critical crossover threshold occurs (\(p \le 0.001\)). * Above 1,500 tokens: Verified search surpasses the monolith, reaching ~44% accuracy at the highest tiers compared to ~40% for the monolith.

为大语言模型添加推理结构（如规划、验证和修正循环）可以使其进行搜索和自我纠错，但这些过程会消耗宝贵的Token预算。本文旨在探究是否存在这样一个Token预算阈值：低于该阈值时，结构化开销会损害性能；高于该阈值时，结构化开销则能提升性能。

研究人员在FinQA和TAT-QA财务推理任务上评估了两类系统（单一整体LLM vs. 验证搜索架构），跨越了从250到42,000个Token的14个预算档位，并发现了明显的性能交叉点： * 1,000 Token以下： 结构化开销挤占了模型的响应空间，导致验证搜索表现不佳。 * 1,000至1,500 Token之间： 出现关键的交叉阈值（\(p \le 0.001\)）。 * 1,500 Token以上： 验证搜索超越单一整体模型，在最高预算档位达到约44%的准确率，而单一整体模型约为40%。

📖 Abstract

Adding inference structure to a language model lets it search, verify, and revise, but these actions consume the very budget they are supposed to use well. In this paper, we investigate whether there exists a token-budget threshold, below which the overhead of planning and verification hurts performance and above which it helps.

We evaluate two systems on FinQA and TAT-QA financial reasoning tasks, using GPT-5.4 mini across 14 budget tiers ranging from 250 to 42,000 output-equivalent tokens. The first system is a monolith, which is a single LLM call. The second is a verified search architecture that adds planning, label-blind checking, and repair capabilities. We run 1,000 cases for a total of 28,000 completed cells.

Both systems score 0% at the two lowest tiers, where neither can fit a complete prompt. At 1,000 tokens, the monolith reaches 18% accuracy while verified search scores near 0%, since the planning overhead leaves no room for an answer. From 1,500 tokens onward, verified search surpasses the monolith and maintains a consistent advantage, reaching approximately 44% at the highest tiers while the monolith reaches approximately 40%. The crossover occurs between 1,000 and 1,500 output-equivalent tokens, confirmed by a strict intersection-union test (\(p \le 0.001\) at both endpoints).

为语言模型添加推理结构使其能够进行搜索、验证和修正，但这些操作消耗了它们本应用来优化输出的预算。在本文中，我们探究是否存在这样一个Token预算阈值：低于该阈值时，规划和验证的开销会损害性能；高于该阈值时，则能带来帮助。

我们在FinQA和TAT-QA财务推理任务上评估了两类系统，使用 GPT-5.4 mini 跨越了从250到42,000个等效输出Token的14个预算档位。第一类系统是单一整体模型（monolith），即单次LLM调用；第二类是验证搜索架构（verified search architecture），增加了规划、标签盲检（label-blind checking）和修复能力。我们运行了1,000个测试用例，总计完成了28,000个实验单元。

在预算最低的两个档位中，由于两个系统都无法容纳完整的提示词，其得分均为0%。在1,000 Token时，单一模型达到了18%的准确率，而验证搜索的得分接近0%，因为规划开销没有留下任何回答的空间。从1,500 Token开始，验证搜索超越了单一模型并保持了持续的优势，在最高档位达到了约44%的准确率，而单一模型约为40%。交叉点发生在1,000到1,500个等效输出Token之间，并通过严格的交集-并集检验得到了证实（两端终点的 \(p \le 0.001\)）。

📋 Metadata & Additional Information

🔗 Full-Text & Access Links

🔗 全文与访问链接