跳转至

文章背景与核心概要

在大语言模型（LLM）的实际部署与服务中，由于需要将序列填充（pad）至固定的最大长度，往往会造成严重的计算资源浪费并降低系统吞吐量。为了解决这一痛点，提前预测模型的输出长度能够启用“长度感知调度”（length-aware scheduling），从而显著减少填充开销，这在长文本推理和强化学习应用中尤为重要。然而，传统方法主要依赖基于Token的熵（如熵引导的Token池化），却常常忽略了不同Token之间语义内容的变化，导致关键Token权重不足而信息稀疏的Token被赋予过高权重，影响了预测的可靠性。

为了克服这一局限，本文作者推出了 ESTP（Entropy-and-Semantic Token Pooling，熵与语义Token池化） 这一轻量级框架。该框架将熵与直接从LLM预填充（prefill）阶段自注意力权重中提取的基于注意力的重要性得分相结合。由于ESTP能够直接复用预填充激活值，因此几乎不会带来额外的内存开销和延迟。在 ForeLen 基准测试中，该方法在准确率和错误率方面均优于现有的基线模型，为长度感知的大模型服务系统提供了一个实用且高效的构建模块。

When Entropy Is Not Enough: Reclaiming Lost Semantics in LLM Output Length Prediction

When Entropy Is Not Enough: Reclaiming Lost Semantics in LLM Output Length Prediction

arXiv ID: arXiv:2608.15592 [cs.AI]
Submitted: August 16, 2026

arXiv ID: arXiv:2608.15592 [cs.AI]
Submitted: August 16, 2026

Authors: Feiyang Ren, Shengtao Wen, Lingbing Guo, Yu Tian, Yuanning Cui, Xiang Chen

Authors: Feiyang Ren, Shengtao Wen, Lingbing Guo, Yu Tian, Yuanning Cui, Xiang Chen

📋 Summary

高效的大语言模型（LLM）服务往往受限于将序列填充至固定最大长度的需求，这会消耗计算资源并降低吞吐量。提前预测输出长度使得采用长度感知调度成为可能，从而减少开销。这一优势在长文本推理和强化学习应用中尤为明显。传统方法主要依赖基于Token的熵（例如熵引导的Token池化）作为主要信号，但它们往往忽略了不同Token之间语义内容的差异。因此，重要的Token经常被赋予较低的权重，而承载较少信息的Token则受到不相称的重构强调，这损害了长度预测的可靠性。为此，作者引入了 ESTP（Entropy-and-Semantic Token Pooling，熵与语义Token池化），这是一个轻量级框架，通过将熵与基于注意力的重要性得分相结合来解决这个问题。这些得分直接来源于LLM预填充阶段计算出的自注意力权重，这使得ESTP能够以极少的额外计算捕获不确定性和语义重要性。由于该框架复用预填充激活值，它几乎不增加额外的内存开销，并仅引入极小的延迟。在 ForeLen 基准测试中，ESTP 优于基线方法，在多数场景下实现了更好的预测准确率和更低的错误率。当在端到端系统测试中与长度感知调度器集成时，它进一步帮助提高了整体吞吐量并降低了填充率。我们的结果为长度感知的LLM服务系统提供了一个实用且高效的构建模块。

📋 Summary

Efficiently serving Large Language Models (LLMs) is often bottlenecked by the requirement to pad sequences to a fixed maximum length, which wastes compute resources and lowers throughput. To mitigate this, predicting output lengths ahead of time enables length-aware scheduling, significantly reducing padding overhead—especially in long-context reasoning and reinforcement learning applications.

Traditional approaches rely primarily on token-wise entropy (e.g., entropy-guided token pooling), but they frequently overlook variations in semantic content across tokens. Consequently, crucial tokens are underweighted while information-sparse tokens receive disproportionate emphasis, hindering prediction reliability.

To overcome this, the authors introduce ESTP (Entropy-and-Semantic Token Pooling), a lightweight framework that pairs entropy with attention-based importance scores derived directly from self-attention weights during the LLM prefill phase. Because ESTP reuses prefill activations, it incurs virtually zero extra memory overhead and minimal latency, outperforming baseline models on the ForeLen benchmark in accuracy and error reduction.

📄 Abstract

Efficient LLM serving is often bottlenecked by the need to pad sequences to a fixed maximum length, and this wastes compute and degrades throughput. Predicting output lengths in advance makes it possible to adopt length-aware scheduling, and this reduces the overhead. This advantage is especially pronounced in long-context reasoning and reinforcement learning applications. Existing approaches, such as entropy-guided token pooling, use token-wise entropy as their primary signal, but they tend to ignore differences in semantic content across tokens. So, important tokens are often underweighted, and tokens carrying little information receive disproportionate emphasis. This hurts the reliability of length prediction. We introduce ESTP (Entropy-and-Semantic Token Pooling), a lightweight framework that addresses this issue by combining entropy with attention-based importance scores. These scores are derived directly from the self-attention weights computed during the LLM prefill phase, and this allows ESTP to capture both uncertainty and semantic importance with minimal additional computation. Since the framework reuses prefill activations, it adds almost no extra memory overhead and introduces only minimal latency. On the ForeLen benchmark, ESTP outperforms baseline methods, achieves better prediction accuracy and lower error rates in most scenarios. When integrated with a length-aware scheduler in end-to-end system tests, it further helps improve overall throughput and reduce the padding ratio. Our results offer a practical and effective building block for length-aware LLM serving systems.

🔗 Additional Resources

(许可证图标参考：license icon 知识共享署名 4.0 国际许可协议)

🔗 Additional Resources

(License Icon Reference: license icon Creative Commons Attribution 4.0 International)