跳转至

思维链何时有效、何时失效：大模型推理中串行深度瓶颈的实证研究

文章背景与核心概要

长期以来，业界普遍认为思维链（Chain-of-Thought, CoT）提示词能够普遍提升大语言模型（LLM）的推理能力。本文基于 \(H_{dp}\) 带宽边界框架，对这一假设进行了深入的实证审视，指出 CoT 并非通用的推理增强器，而是一种“带宽旁路”（bandwidth bypass），主要用于将超出 Transformer 单次前向传递（single-pass）能力的串行计算外化。

通过对多个指令微调模型和基准测试的实证评估，研究揭示了清晰的串行深度梯度：高深度的 \(\mathbf{P}\)-完全任务（如 GSM8K、MATH）从 CoT 中获益巨大；浅层的 \(\mathbf{TC^0}\) 任务（如 MMLU、ARC）发现 CoT 在结构上是冗余的；而中间类任务（如 HumanEval）则表现出依赖于模型规模的转变。这些发现表明，CoT 仅对受限于单次传递容量的串行计算有帮助，而对于原本就能容纳的任务则显得多余。

Summary

This paper investigates the common assumption that Chain-of-Thought (CoT) prompting universally improves Large Language Model (LLM) reasoning. Using the \(H_{dp}\) bandwidth bound framework, the study demonstrates that CoT acts as a "bandwidth bypass" rather than a universal reasoning enhancer. It externalizes serial computation that exceeds a transformer's single-pass capacity.

The empirical evaluation across multiple instruction-tuned models and benchmarks reveals a clear serial-depth gradient: * High-depth \(\mathbf{P}\)-complete tasks (e.g., GSM8K, MATH) benefit massively from CoT (+54 to +68 pp recovery gap). * Shallow \(\mathbf{TC^0}\) tasks (e.g., MMLU, ARC) find CoT structurally redundant with negligible impact. * Intermediate tasks (e.g., HumanEval) show a model-size-dependent transition.

This paper investigates the common assumption that Chain-of-Thought (CoT) prompting universally improves Large Language Model (LLM) reasoning. Using the \(H_{dp}\) bandwidth bound framework, the study demonstrates that CoT acts as a "bandwidth bypass" rather than a universal reasoning enhancer. It externalizes serial computation that exceeds a transformer's single-pass capacity.

The empirical evaluation across multiple instruction-tuned models and benchmarks reveals a clear serial-depth gradient: * High-depth \(\mathbf{P}\)-complete tasks (e.g., GSM8K, MATH) benefit massively from CoT (+54 to +68 pp recovery gap). * Shallow \(\mathbf{TC^0}\) tasks (e.g., MMLU, ARC) find CoT structurally redundant with negligible impact. * Intermediate tasks (e.g., HumanEval) show a model-size-dependent transition.

Document Metadata

字段 详情
arXiv ID arXiv:2608.09942 [cs.CL]
主学科 Computation and Language (cs.CL)
辅学科 Artificial Intelligence (cs.AI), Machine Learning (cs.LG)
作者 Tughanbulut Kurtulush
提交日期 June 23, 2026
预注册 OSF: 92jdk
数据与代码 OSF: hteuj
许可协议 license icon Creative Commons Attribution 4.0
Field Details
arXiv ID arXiv:2608.09942 [cs.CL]
Primary Subject Computation and Language (cs.CL)
Secondary Subjects Artificial Intelligence (cs.AI), Machine Learning (cs.LG)
Author Tughanbulut Kurtulush
Submission Date June 23, 2026
Pre-registration OSF: 92jdk
Data and Code OSF: hteuj
License license icon Creative Commons Attribution 4.0

Abstract

人们普遍认为思维链（CoT）提示词能够无条件地提升 LLM 的推理能力。我们通过 \(H_{dp}\) 带宽边界的概念框架（Chen 等人，2024 年）对此进行了研究：尽管形式上的边界仅在渐进意义上（在极大的提示长度下）成立，但它识别出了一个真实的架构瓶颈——超出 Transformer 单次处理容量的串行计算必须被外化，而这正是 CoT 所做的事情。

我们的核心发现在于基准测试内部存在串行深度梯度：单次传递（无 CoT）的准确率随着每个样本的串行深度呈单调下降，而 CoT 在大致上对深度不敏感。我们在三个指令微调模型（Qwen-2.5-7B/32B、Llama-3.1-8B）和五个标准 NLP 基准上，在实际的上下文长度下测量了 CoT 的效果。

  • 在高深度的 \(\mathbf{P}\)-完全任务（GSM8K、MATH）上，所有模型的 CoT 恢复差距（recovery gap）均达到 +54 到 +68 个百分点（pp）。
  • 在浅层的 \(\mathbf{TC^0}\) 任务（MMLU、ARC）上，CoT 在结构上是冗余的（\(\Delta\) 在 \([0.0, +4.6]\) pp 之间，没有显著的负面影响）——尽管无 CoT 的高基线（在 ARC 上高达 95%）可能反映了数据污染，因此这一零结果并非一个干净的架构测试。
  • 中间类任务 \(\mathbf{L}\)（HumanEval）表现出依赖于模型规模的转变：+23.2 pp (32B)、+9.1 pp (8B)、-28.7 pp (7B)。

跨基准的深度恢复相关性为 Spearman \(\rho = 0.661\)（\(p = 0.007\)，\(n = 15\)）；在进行 Bonferroni 校正后，15 个基准级别的 McNemar 检验中有 9 个显著。我们的结果在 OSF 上进行了预注册，表明 CoT 并非通用的推理增强器，而是充当了一种带宽旁路：它有助于处理那些压榨单次传递容量的串行计算，而对于已经能够适配的任务则是冗余的。

It is widely assumed that chain-of-thought (CoT) prompting universally improves LLM reasoning. We investigate this through the conceptual framework of the \(H_{dp}\) bandwidth bound (Chen et al., 2024): although the formal bound binds only asymptotically (at astronomically large prompt lengths), it identifies a real architectural bottleneck -- serial computation exceeding a transformer's single-pass capacity must be externalised, which is what CoT does.

Our central finding is a within-benchmark serial-depth gradient: single-pass (no-CoT) accuracy degrades monotonically with per-item serial depth, while CoT is approximately depth-invariant. We measure CoT effects across three instruction-tuned models (Qwen-2.5-7B/32B, Llama-3.1-8B) and five standard NLP benchmarks at practical context lengths.

  • On high-depth \(\mathbf{P}\)-complete tasks (GSM8K, MATH), CoT gives a +54 to +68 pp recovery gap across all models.
  • On shallow \(\mathbf{TC^0}\) tasks (MMLU, ARC), CoT is structurally redundant (\(\Delta\) in \([0.0, +4.6]\) pp, no significant negative effect) -- though high no-CoT baselines (up to 95% on ARC) may reflect contamination, so this null is not a clean architectural test.
  • The intermediate class \(\mathbf{L}\) (HumanEval) shows a model-size-dependent transition: +23.2 pp (32B), +9.1 pp (8B), -28.7 pp (7B).

The cross-benchmark depth-recovery correlation is Spearman \(\rho = 0.661\) (\(p = 0.007\), \(n = 15\)); 9 of 15 benchmark-level McNemar tests are significant after Bonferroni correction. Pre-registered on OSF, our results indicate that CoT is not a universal reasoning enhancer but acts as a bandwidth bypass: it helps serial computation that strains single-pass capacity and is redundant for tasks that already fit.

Full-Text & Reference Links