文章背景与核心概要
大语言模型（LLM）的轻量化与压缩一直是AI领域的核心研究方向之一。层剪枝（Layer Pruning）作为一种经典的结构化压缩技术，此前在分类任务中表现优异，往往无需微调或仅需微调即可保留大部分性能。然而，由 Safal Shrestha 等人发表的这篇最新研究表明，层剪枝在面对生成式推理任务（如数学推理 GSM8K、代码生成 HumanEval+）时，会导致模型性能出现灾难性下降。
论文的核心发现指出，层剪枝不仅仅造成了表面文本生成的退化，更导致了底层关键“算法能力”的根本性丢失，例如精确的算术计算和结构化语法（如括号匹配）的生成。通过在现实的后训练约束（如单张 80GB GPU）下进行广泛实验，作者发现即使采用任务对齐的微调策略，或者在剪枝后使用近 1000 亿个 Token 进行训练，这种性能差距依然无法弥补。该研究为我们在受限的后训练范式下如何有效运用深度缩减技术提供了重要的警示与指导。
On the Limits of Layer Pruning for Generative Reasoning in Large Language Models
arXiv: 2602.01997
Authors: Safal Shrestha, Anubhav Shrestha, Aadim Nepal, Minwu Kim, Keith Ross
Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
Submitted: 2 Feb 2026 (v1); Last Revised: 6 Aug 2026 (v4)
arXiv: 2602.01997
Authors: Safal Shrestha, Anubhav Shrestha, Aadim Nepal, Minwu Kim, Keith Ross
Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
Submitted: 2 Feb 2026 (v1); Last Revised: 6 Aug 2026 (v4)
Summary
摘要 While layer pruning is a successful strategy for compressing Large Language Models (LLMs) in classification tasks, this paper demonstrates that it significantly degrades performance in generative reasoning tasks (e.g., GSM8K, HumanEval+). The authors find that pruning causes a fundamental loss of algorithmic capabilities—such as arithmetic computation and structural syntax generation—that cannot be easily recovered through standard fine-tuning. Even when using task-aligned recovery or training on large post-pruning datasets, the performance gap persists, suggesting inherent limitations in depth-reduction techniques for complex reasoning.
尽管层剪枝是压缩大语言模型（LLM）用于分类任务的成功策略，但本文证明它显著降低了生成式推理任务（例如 GSM8K、HumanEval+）的性能。作者发现，剪枝会导致算法能力的根本性丧失——例如算术计算和结构化语法生成——而这些能力无法通过标准微调轻易恢复。即使在使用任务对齐的恢复方法或在大型剪枝后数据集上进行训练时，性能差距依然存在，这表明深度缩减技术在处理复杂推理时存在内在局限性。
Abstract
摘要 Recent work has show that layer pruning can effectively compress large language models (LLMs) while retaining strong performance on classification benchmarks, often with little or no finetuning. In contrast, generative reasoning tasks, such as GSM8K and HumanEval\textsuperscript{+}, exhibit substantially weaker recovery.
近期的研究表明，层剪枝可以有效压缩大语言模型（LLM），同时在分类基准测试中保持强劲的性能，通常只需很少甚至不需要微调。相比之下，诸如 GSM8K 和 HumanEval\textsuperscript{+} 等生成式推理任务展现出的恢复能力则要弱得多。
We show that beyond surface-level text degradation, pruning leads to a loss of key algorithmic capabilities, including arithmetic computation and balanced parenthesis generation. Under realistic post-training constraints, using a single 80GB GPU and without access to pretraining-scale data or compute, we evaluate a simple recovery strategy based on supervised finetuning with self-generated responses. This approach recovers up to 90\% of baseline performance on classification tasks, but recovery for generative reasoning remains limited.
我们表明，除了表面层面的文本退化之外，剪枝还会导致关键算法能力的丧失，包括算术计算和平衡括号生成。在现实的后训练约束下（使用单张 80GB GPU，且无法获取预训练规模的数据或算力），我们评估了一种基于使用自生成响应进行监督微调的简单恢复策略。这种方法在分类任务上恢复了高达 90% 的基线性能，但对生成式推理的恢复仍然十分有限。
We further find that this gap persists even under a favorable task-aligned recovery setting, where pruned models are fully finetuned on self-generated GSM8K responses, suggesting that the degradation is not merely due to generic instruction data or parameter-efficient tuning. As complementary evidence, we analyze a depth-pruned model trained with nearly 100B post-pruning tokens and find that deficits persist even on simple arithmetic tasks that do not require multi-step generation. Overall, we characterize practical recovery limits of layer pruning for generative reasoning and provide guidance on when depth reduction is effective under constrained post-training regimes.
我们进一步发现，即使在有利的任务对齐恢复设置下（其中剪枝模型在自生成的 GSM8K 响应上进行了完全微调），这种差距依然存在，这表明性能退化不仅仅是因为通用的指令数据或参数高效微调。作为补充证据，我们分析了一个使用近 1000 亿个剪枝后 Token 训练的深度剪枝模型，发现即使在不需要多步生成的简单算术任务上，缺陷依然存在。总体而言，我们刻画了层剪枝用于生成式推理的实际恢复极限，并为在受限后训练机制下何时有效进行深度缩减提供了指导。
Accessing the Paper
获取论文 * View PDF * HTML (Experimental) * TeX Source * DOI: https://doi.org/10.48550/arXiv.2602.01997
Submission History
提交历史 * v1: 2 Feb 2026 * v2: 10 Apr 2026 * v3: 4 Aug 2026 * v4: 6 Aug 2026
- v1: 2026年2月2日
- v2: 2026年4月10日
- v3: 2026年8月4日
- v4: 2026年8月6日